Oct 30 05:52:41.612477 ip-10-0-131-226 systemd[1]: Starting Kubernetes Kubelet... Oct 30 05:52:42.043563 ip-10-0-131-226 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:52:42.043563 ip-10-0-131-226 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 05:52:42.043563 ip-10-0-131-226 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:52:42.043563 ip-10-0-131-226 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:52:42.047157 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047082 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:42.047157 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047156 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047160 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047164 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047167 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047170 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047172 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047176 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047179 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047181 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047184 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047186 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047189 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047192 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047194 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047197 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047199 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047202 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047205 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047207 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047223 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:42.047236 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047230 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047233 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047235 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047238 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047241 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047244 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047247 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047250 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047253 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047256 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047258 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047261 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047263 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047266 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047269 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047271 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047276 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047280 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047283 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:42.047690 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047286 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047288 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047291 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047293 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047296 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047298 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047301 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047304 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047306 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047309 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047311 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047314 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047317 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047320 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047322 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047326 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047328 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047332 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047335 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:42.048159 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047338 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047341 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047343 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047346 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047349 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047351 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047353 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047356 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047359 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047361 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047364 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047366 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047369 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047371 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047374 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047376 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047379 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047381 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047384 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:42.048620 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047391 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047394 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047396 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047399 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047402 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047405 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047407 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047410 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047413 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047415 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047418 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047420 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047423 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047426 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047428 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047431 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047433 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047436 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047438 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:42.049056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047441 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047446 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047449 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047452 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047455 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047457 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047460 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047462 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.047465 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048249 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048257 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048260 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048263 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048266 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048270 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048273 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048275 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048278 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048282 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:42.049526 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048284 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048287 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048290 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048292 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048294 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048297 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048300 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048302 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048305 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048308 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048310 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048313 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048316 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048318 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048321 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048323 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048326 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048329 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048332 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:42.049981 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048335 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048339 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048343 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048347 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048349 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048352 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048355 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048357 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048360 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048363 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048366 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048368 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048372 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048376 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048379 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048382 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048385 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048387 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048390 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048392 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:42.050456 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048395 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048397 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048400 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048402 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048405 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048408 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048411 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048414 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048416 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048419 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048422 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048425 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048428 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048430 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048433 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048435 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048438 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048440 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048443 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048445 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:42.051047 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048448 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048451 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048454 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048457 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048460 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048462 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048465 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048467 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048469 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048472 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048474 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048477 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048480 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048482 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048485 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048487 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048490 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048492 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048495 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048497 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:42.051538 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048500 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048502 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048505 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048507 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048510 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048513 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048515 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048518 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048520 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048523 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048525 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048528 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048531 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048534 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048536 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048539 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.048542 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049001 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049005 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:42.052016 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049008 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049011 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049014 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049016 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049019 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049023 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049025 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049028 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049031 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049034 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049036 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049039 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049042 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049046 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049050 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049053 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049057 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049060 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049063 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:42.052492 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049065 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049068 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049071 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049074 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049077 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049079 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049082 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049085 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049087 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049091 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049093 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049096 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049099 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049101 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049104 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049106 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049109 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049112 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049116 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:42.052932 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049120 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049123 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049126 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049129 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049132 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049134 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049137 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049139 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049142 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049145 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049147 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049150 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049152 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049155 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049157 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049160 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049163 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049165 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049167 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:42.053389 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049170 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049173 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049178 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049180 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049183 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049186 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049189 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049192 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049194 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049197 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049199 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049202 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049204 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049207 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049226 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049229 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049231 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049234 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049237 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049239 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:42.053879 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049242 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049244 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049247 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049249 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049252 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049254 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049257 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049259 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049262 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049265 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049267 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049270 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049272 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049275 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049278 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049281 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049283 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049286 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049289 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049293 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:42.054355 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049295 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049298 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049300 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049303 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049305 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049308 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.049311 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050288 2575 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050299 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050305 2575 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050311 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050315 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050318 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050322 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050327 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050330 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050333 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050337 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050340 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050343 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050346 2575 flags.go:64] FLAG: --cgroup-root="" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050349 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050352 2575 flags.go:64] FLAG: --client-ca-file="" Oct 30 05:52:42.054832 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050355 2575 flags.go:64] FLAG: --cloud-provider="external" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050358 2575 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050362 2575 flags.go:64] FLAG: --cluster-domain="" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050368 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050371 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050375 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050378 2575 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050382 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050386 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050389 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050392 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050395 2575 flags.go:64] FLAG: --contention-profiling="false" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050398 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050401 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050405 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050408 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050413 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050416 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050419 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050422 2575 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050425 2575 flags.go:64] FLAG: --enable-server="true" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050428 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050432 2575 flags.go:64] FLAG: --event-burst="100" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050435 2575 flags.go:64] FLAG: --event-qps="50" Oct 30 05:52:42.055417 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050438 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050441 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050444 2575 flags.go:64] FLAG: --eviction-hard="" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050448 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050452 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050455 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050459 2575 flags.go:64] FLAG: --eviction-soft="" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050462 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050465 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050468 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050471 2575 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050475 2575 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050478 2575 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050481 2575 flags.go:64] FLAG: --feature-gates="" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050485 2575 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050488 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050492 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050496 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050499 2575 flags.go:64] FLAG: --healthz-port="10248" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050502 2575 flags.go:64] FLAG: --help="false" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050504 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050508 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050510 2575 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 05:52:42.055990 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050513 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050516 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050520 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050523 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050525 2575 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050528 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050531 2575 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050534 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050537 2575 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050539 2575 flags.go:64] FLAG: --kube-reserved="" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050542 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050545 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050548 2575 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050551 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050554 2575 flags.go:64] FLAG: --lock-file="" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050557 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050559 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050562 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050568 2575 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050571 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050574 2575 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050576 2575 flags.go:64] FLAG: --logging-format="text" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050579 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050583 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 05:52:42.056569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050586 2575 flags.go:64] FLAG: --manifest-url="" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050589 2575 flags.go:64] FLAG: --manifest-url-header="" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050594 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050597 2575 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050601 2575 flags.go:64] FLAG: --max-pods="110" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050604 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050607 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050610 2575 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050613 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050616 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050619 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050621 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050628 2575 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050631 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050634 2575 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050637 2575 flags.go:64] FLAG: --pod-cidr="" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050640 2575 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050643 2575 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050646 2575 flags.go:64] FLAG: --pods-per-core="0" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050648 2575 flags.go:64] FLAG: --port="10250" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050651 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050654 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c34d0aee64d866a3" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050657 2575 flags.go:64] FLAG: --qos-reserved="" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050660 2575 flags.go:64] FLAG: --read-only-port="10255" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050663 2575 flags.go:64] FLAG: --register-node="true" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050666 2575 flags.go:64] FLAG: --register-with-taints="" Oct 30 05:52:42.057486 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050673 2575 flags.go:64] FLAG: --registry-burst="10" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050676 2575 flags.go:64] FLAG: --registry-qps="5" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050679 2575 flags.go:64] FLAG: --reserved-cpus="" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050682 2575 flags.go:64] FLAG: --reserved-memory="" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050686 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050689 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050692 2575 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050695 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050697 2575 flags.go:64] FLAG: --runonce="false" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050701 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050704 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050707 2575 flags.go:64] FLAG: --seccomp-default="false" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050710 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050712 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050716 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050719 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050723 2575 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050725 2575 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050728 2575 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050731 2575 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050734 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050737 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050740 2575 flags.go:64] FLAG: --system-cgroups="" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050743 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050748 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 05:52:42.058594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050751 2575 flags.go:64] FLAG: --tls-cert-file="" Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050753 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050758 2575 flags.go:64] FLAG: --tls-min-version="" Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050762 2575 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050765 2575 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050768 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050771 2575 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050773 2575 flags.go:64] FLAG: --v="2" Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050778 2575 flags.go:64] FLAG: --version="false" Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050782 2575 flags.go:64] FLAG: --vmodule="" Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050786 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.050789 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050927 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050930 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050934 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050936 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050939 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050943 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050945 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050948 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050950 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050953 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:42.059723 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050956 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050958 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050961 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050963 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050965 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050968 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050971 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050973 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050976 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050978 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050981 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050983 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050986 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050990 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050994 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.050997 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051000 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051002 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051005 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:42.060676 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051008 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051010 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051013 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051015 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051018 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051021 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051024 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051026 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051029 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051032 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051035 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051037 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051040 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051042 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051045 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051047 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051050 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051053 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051056 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051058 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:42.061430 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051061 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051064 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051066 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051069 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051071 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051074 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051077 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051081 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051084 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051086 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051089 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051092 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051094 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051097 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051099 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051102 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051105 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051107 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051110 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051113 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:42.062052 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051115 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051118 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051121 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051123 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051126 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051128 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051131 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051134 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051138 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051142 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051145 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051147 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051150 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051152 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051155 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051157 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051160 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051164 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051169 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:42.062580 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051174 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051177 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051179 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051182 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051185 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051187 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051190 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051192 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051195 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051198 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051200 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051203 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051206 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051222 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051225 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051227 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051230 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:42.063056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051234 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.051261 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051295 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051300 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051303 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051305 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051308 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051310 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051313 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051316 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051319 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051321 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051324 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051326 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051331 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051333 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:42.063528 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051338 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051340 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051343 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051345 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051348 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051350 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051353 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051356 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051358 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051361 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051364 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051366 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051369 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051371 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051374 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051376 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051379 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051382 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051385 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:42.064056 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051387 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051390 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051393 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051395 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051398 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051400 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051403 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051405 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051408 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051410 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051413 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051417 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051419 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051423 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051425 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051428 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051431 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051435 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051438 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051441 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:42.064859 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051443 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051446 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051448 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051451 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051454 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051456 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051460 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051465 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051468 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051471 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051474 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051477 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051479 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051482 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051484 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051487 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051489 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051492 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051494 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:42.065376 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051497 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051500 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051502 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051505 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051509 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051511 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051516 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051519 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051521 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051524 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051527 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051529 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051532 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051534 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051537 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051540 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051542 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051545 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051548 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051550 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:42.065967 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051553 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051555 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051558 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051560 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051563 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051566 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051568 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051571 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051573 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051576 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051579 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051582 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051585 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.051588 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.059173 2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.059189 2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.059234 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.059242 2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059326 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:42.066470 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059334 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059340 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059345 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059350 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059354 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059359 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059363 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059367 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059371 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059375 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059380 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059385 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059389 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059394 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059398 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059403 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059407 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059412 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059416 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059421 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:42.066988 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059425 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059430 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059434 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059439 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059443 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059448 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059452 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059456 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059460 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059465 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059470 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059474 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059478 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059482 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059486 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059490 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059495 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059499 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059503 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:42.067546 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059508 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059514 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059520 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059525 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059529 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059533 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059538 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059544 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059551 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059556 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059561 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059566 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059570 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059574 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059578 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059583 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059588 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059592 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059596 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:42.068057 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059600 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059604 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059609 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059614 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059618 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059623 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059627 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059632 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059636 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059641 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059645 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059649 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059653 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059658 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059663 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059668 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059672 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059676 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059681 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059685 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059690 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:42.068519 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059694 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059698 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059703 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059707 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059712 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059716 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059720 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059724 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059728 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059732 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059736 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059740 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059744 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059749 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059753 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059758 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059762 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059766 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059770 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059775 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:42.069104 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059779 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059783 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059788 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059792 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059796 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.059800 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.059844 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060056 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060063 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060068 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060073 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060077 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060081 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060085 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060090 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060095 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:42.069919 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060099 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060104 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060108 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060113 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060117 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060122 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060126 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060130 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060134 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060139 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060143 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060147 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060152 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060156 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060161 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060165 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060169 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060174 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060178 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060182 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:42.070558 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060186 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060190 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060194 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060199 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060203 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060207 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060229 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060233 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060237 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060241 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060246 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060250 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060254 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060258 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060263 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060267 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060271 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060276 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060282 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:42.071413 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060289 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060294 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060298 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060302 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060307 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060311 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060315 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060319 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060324 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060328 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060332 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060336 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060340 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060345 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060349 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060353 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060357 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060362 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060366 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060371 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:42.072106 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060375 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060379 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060383 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060387 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060392 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060396 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060399 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060404 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060412 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060419 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060423 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060428 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060433 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060437 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060441 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060446 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060450 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060454 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060459 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:42.072635 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060463 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060467 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060471 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060476 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060480 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060484 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060488 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060492 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060496 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060501 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060505 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060509 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060513 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060518 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060522 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060526 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060530 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060534 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:42.073074 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060538 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.060576 2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060630 2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060637 2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060642 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060647 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060651 2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060656 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060662 2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060668 2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060673 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060677 2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060681 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060684 2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060688 2575 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060692 2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:52:42.073560 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060696 2575 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060701 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060705 2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060709 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060713 2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060717 2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060721 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060725 2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060729 2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060734 2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060738 2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060742 2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060747 2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060751 2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060755 2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060759 2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060764 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060768 2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060772 2575 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060776 2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:52:42.073937 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060781 2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060789 2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060795 2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060800 2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060804 2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060808 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060813 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060817 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060822 2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060826 2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060830 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060834 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060839 2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060843 2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060847 2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060851 2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060855 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060860 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060864 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:52:42.074520 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060868 2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060872 2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060876 2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060881 2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060885 2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060889 2575 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060893 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060897 2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060901 2575 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060905 2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060910 2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060914 2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060918 2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060925 2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060930 2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060941 2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060945 2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060949 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060953 2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060957 2575 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:52:42.074987 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060961 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060966 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060970 2575 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060974 2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060979 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060983 2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060987 2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060991 2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.060995 2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061000 2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061005 2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061009 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061013 2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061017 2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061022 2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061028 2575 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061032 2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061036 2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061040 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061045 2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:52:42.075509 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061049 2575 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061053 2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061057 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061061 2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061065 2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061071 2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061076 2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061080 2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061085 2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061089 2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061093 2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061097 2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:42.061101 2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.061173 2575 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.065584 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.066662 2575 server.go:1014] "Starting client certificate rotation" Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.066764 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.067603 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:52:42.075979 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.069498 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:52:42.076414 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.074963 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:52:42.085906 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.085890 2575 log.go:25] "Validated CRI v1 runtime API" Oct 30 05:52:42.089056 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.089043 2575 log.go:25] "Validated CRI v1 image API" Oct 30 05:52:42.089904 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.089884 2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 05:52:42.092702 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.092677 2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8487a56f-bfb9-49be-a510-44df142a7f11:/dev/nvme0n1p4 c8c7bd7a-512a-420c-8080-3ad6977584cd:/dev/nvme0n1p3] Oct 30 05:52:42.092746 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.092702 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 05:52:42.096828 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.096729 2575 manager.go:217] Machine: {Timestamp:2026-10-30 05:52:42.096265744 +0000 UTC m=+0.359919124 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095819 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24b653546c15a3f10a5fc601b9fdc8 SystemUUID:ec24b653-546c-15a3-f10a-5fc601b9fdc8 BootID:4b0ac8de-d962-4a3c-88ac-44e919a117d4 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fb:3d:46:79 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fb:3d:46:79 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:b3:54:dc:9a:e9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 05:52:42.096828 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.096824 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 05:52:42.096950 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.096917 2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 05:52:42.100464 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.100438 2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 05:52:42.100594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.100467 2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-226.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 05:52:42.100639 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.100596 2575 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 05:52:42.100639 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.100604 2575 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 05:52:42.100639 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.100629 2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:52:42.100715 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.100646 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:52:42.101450 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.101440 2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 05:52:42.105370 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.105358 2575 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 05:52:42.105512 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.105502 2575 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 05:52:42.106349 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.106339 2575 kubelet.go:493] "Attempting to sync node with API server" Oct 30 05:52:42.106380 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.106357 2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 05:52:42.106380 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.106376 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 05:52:42.106436 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.106388 2575 kubelet.go:405] "Adding apiserver pod source" Oct 30 05:52:42.106436 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.106401 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 05:52:42.106715 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.106704 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:52:42.108886 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.108870 2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 05:52:42.113948 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.113935 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:52:42.114930 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.114916 2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 05:52:42.114968 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.114962 2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 05:52:42.115106 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.115096 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 05:52:42.115135 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.115112 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 05:52:42.115135 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.115119 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 05:52:42.115135 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.115124 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 05:52:42.115135 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.115130 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 05:52:42.115135 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.115135 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 05:52:42.115871 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.115863 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 05:52:42.115898 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.115879 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 05:52:42.115898 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.115887 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 05:52:42.115898 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.115892 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 05:52:42.115974 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.115904 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 05:52:42.115974 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.115913 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 05:52:42.116724 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.116714 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 05:52:42.116753 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.116728 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 05:52:42.120315 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.120302 2575 server.go:1263] "Started kubelet" Oct 30 05:52:42.120442 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.120390 2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 05:52:42.120536 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.120422 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 05:52:42.120536 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.120504 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 05:52:42.121031 ip-10-0-131-226 systemd[1]: Started Kubernetes Kubelet. Oct 30 05:52:42.125363 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.125342 2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 05:52:42.126688 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.126664 2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 05:52:42.127007 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.126981 2575 server.go:317] "Adding debug handlers to kubelet server" Oct 30 05:52:42.130697 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.130680 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 05:52:42.131145 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.131132 2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 05:52:42.131253 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.131228 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-226.ec2.internal\" not found" Oct 30 05:52:42.131767 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.131677 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 05:52:42.132382 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.131989 2575 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 05:52:42.132382 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.132003 2575 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 05:52:42.132382 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.132120 2575 factory.go:55] Registering systemd factory Oct 30 05:52:42.132382 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.132134 2575 factory.go:223] Registration of the systemd container factory successfully Oct 30 05:52:42.132382 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.132147 2575 reconstruct.go:97] "Volume reconstruction finished" Oct 30 05:52:42.132382 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.132159 2575 reconciler.go:29] "Reconciler: start to sync state" Oct 30 05:52:42.132382 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.132341 2575 factory.go:153] Registering CRI-O factory Oct 30 05:52:42.132382 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.132358 2575 factory.go:223] Registration of the crio container factory successfully Oct 30 05:52:42.132656 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.132406 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 05:52:42.132656 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.132425 2575 factory.go:103] Registering Raw factory Oct 30 05:52:42.132656 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.132438 2575 manager.go:1199] Started watching for new ooms in manager Oct 30 05:52:42.133438 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.133417 2575 manager.go:319] Starting recovery of all containers Oct 30 05:52:42.143025 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.143008 2575 manager.go:324] Recovery completed Oct 30 05:52:42.146664 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.146652 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:42.149136 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.149122 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:42.149242 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.149147 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:42.149242 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.149157 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:42.149721 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.149708 2575 cpu_manager.go:226] "Starting" policy="none" Oct 30 05:52:42.149721 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.149719 2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 05:52:42.149806 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.149735 2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 05:52:42.149806 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.149720 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:42.150503 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.150490 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:42.150570 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.150512 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:42.150570 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.150540 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:42.151958 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.151946 2575 policy_none.go:50] "Start" Oct 30 05:52:42.151958 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.151960 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 05:52:42.152033 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.151969 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 05:52:42.153976 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.153962 2575 policy_none.go:44] "Start" Oct 30 05:52:42.168341 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.168315 2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 05:52:42.168341 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.168322 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-226.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 05:52:42.169465 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.168320 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-226.ec2.internal.18e338d489d564cb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-226.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-226.ec2.internal,},FirstTimestamp:2026-10-30 05:52:42.120275147 +0000 UTC m=+0.383928527,LastTimestamp:2026-10-30 05:52:42.120275147 +0000 UTC m=+0.383928527,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-226.ec2.internal,}" Oct 30 05:52:42.186789 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.186763 2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 05:52:42.186961 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.186924 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 05:52:42.186961 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.186932 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 05:52:42.187169 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.187154 2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 05:52:42.187353 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.187295 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-226.ec2.internal.18e338d48b8dc435 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-226.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-226.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-226.ec2.internal,},FirstTimestamp:2026-10-30 05:52:42.149135413 +0000 UTC m=+0.412788793,LastTimestamp:2026-10-30 05:52:42.149135413 +0000 UTC m=+0.412788793,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-226.ec2.internal,}" Oct 30 05:52:42.188985 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.188970 2575 manager.go:341] "Starting Device Plugin manager" Oct 30 05:52:42.189059 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.189001 2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 05:52:42.189059 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.189012 2575 server.go:87] "Starting device plugin registration server" Oct 30 05:52:42.189250 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.189238 2575 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 05:52:42.189317 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.189252 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 05:52:42.189372 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.189336 2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 05:52:42.189416 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.189407 2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 05:52:42.189459 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.189415 2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 05:52:42.189701 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.189687 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 05:52:42.189901 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.189886 2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 05:52:42.189966 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.189932 2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-226.ec2.internal\" not found" Oct 30 05:52:42.233976 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.223342 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-226.ec2.internal.18e338d48b8e0355 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-226.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-226.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-226.ec2.internal,},FirstTimestamp:2026-10-30 05:52:42.149151573 +0000 UTC m=+0.412804953,LastTimestamp:2026-10-30 05:52:42.149151573 +0000 UTC m=+0.412804953,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-226.ec2.internal,}" Oct 30 05:52:42.233976 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.225611 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9h2bv" Oct 30 05:52:42.238891 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.238875 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-9h2bv" Oct 30 05:52:42.256790 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.256766 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 05:52:42.258027 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.258004 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 05:52:42.258027 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.258026 2575 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 05:52:42.258110 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.258042 2575 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 05:52:42.258110 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.258071 2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 05:52:42.279431 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.279413 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:52:42.289762 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.289741 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:42.290494 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.290478 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:42.290562 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.290508 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:42.290562 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.290523 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:42.290562 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.290553 2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.346292 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.346253 2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.346292 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.346272 2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-226.ec2.internal\": node \"ip-10-0-131-226.ec2.internal\" not found" Oct 30 05:52:42.358437 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.358414 2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal"] Oct 30 05:52:42.358506 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.358476 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:42.359226 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.359197 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:42.359286 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.359245 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:42.359286 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.359255 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:42.361545 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.361534 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:42.361676 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.361662 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.361712 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.361690 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:42.362270 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.362250 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:42.362386 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.362275 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:42.362386 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.362285 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:42.362659 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.362643 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:42.362734 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.362675 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:42.362734 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.362688 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:42.364827 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.364811 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.364912 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.364842 2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:52:42.365520 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.365494 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:52:42.365603 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.365526 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:52:42.365603 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.365541 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:52:42.389815 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.389799 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-226.ec2.internal\" not found" node="ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.396547 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.396528 2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-226.ec2.internal\" not found" node="ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.411101 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.411077 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-226.ec2.internal\" not found" Oct 30 05:52:42.434717 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.434692 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18e4d93490af52d80822ebfbfe232c01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal\" (UID: \"18e4d93490af52d80822ebfbfe232c01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.434797 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.434725 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18e4d93490af52d80822ebfbfe232c01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal\" (UID: \"18e4d93490af52d80822ebfbfe232c01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.434797 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.434749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b8548329cbac7d7fb7409929225464e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-226.ec2.internal\" (UID: \"0b8548329cbac7d7fb7409929225464e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.512183 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.512155 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-226.ec2.internal\" not found" Oct 30 05:52:42.535343 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.535324 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18e4d93490af52d80822ebfbfe232c01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal\" (UID: \"18e4d93490af52d80822ebfbfe232c01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.535399 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.535348 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b8548329cbac7d7fb7409929225464e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-226.ec2.internal\" (UID: \"0b8548329cbac7d7fb7409929225464e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.535399 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.535363 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18e4d93490af52d80822ebfbfe232c01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal\" (UID: \"18e4d93490af52d80822ebfbfe232c01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.535463 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.535415 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18e4d93490af52d80822ebfbfe232c01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal\" (UID: \"18e4d93490af52d80822ebfbfe232c01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.535463 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.535422 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b8548329cbac7d7fb7409929225464e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-226.ec2.internal\" (UID: \"0b8548329cbac7d7fb7409929225464e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.535463 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.535422 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18e4d93490af52d80822ebfbfe232c01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal\" (UID: \"18e4d93490af52d80822ebfbfe232c01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.612842 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.612791 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-226.ec2.internal\" not found" Oct 30 05:52:42.693942 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.693921 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.699604 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:42.699582 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:52:42.713667 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.713649 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-226.ec2.internal\" not found" Oct 30 05:52:42.814518 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.814494 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-226.ec2.internal\" not found" Oct 30 05:52:42.915079 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:42.915025 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-226.ec2.internal\" not found" Oct 30 05:52:43.015608 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:43.015589 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-226.ec2.internal\" not found" Oct 30 05:52:43.067060 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:43.067040 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 05:52:43.067459 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:43.067162 2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 05:52:43.116658 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:43.116500 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-226.ec2.internal\" not found" Oct 30 05:52:43.217207 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:43.217179 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-226.ec2.internal\" not found" Oct 30 05:52:43.243460 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:43.243423 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:47:43 +0000 UTC" deadline="2028-07-28 17:49:41.973852758 +0000 UTC" Oct 30 05:52:43.243460 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:43.243454 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15299h56m58.730401516s" Oct 30 05:52:43.291101 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:43.291066 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e4d93490af52d80822ebfbfe232c01.slice/crio-8e0e5c1aa27c35c585015e70714593aa7aee54a1ff842315c94d39062be850f9 WatchSource:0}: Error finding container 8e0e5c1aa27c35c585015e70714593aa7aee54a1ff842315c94d39062be850f9: Status 404 returned error can't find the container with id 8e0e5c1aa27c35c585015e70714593aa7aee54a1ff842315c94d39062be850f9 Oct 30 05:52:43.291755 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:43.291735 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8548329cbac7d7fb7409929225464e.slice/crio-3b9d2a677fc04337562bb547eb666046282b130b11a94d933c41537e53849fef WatchSource:0}: Error finding container 3b9d2a677fc04337562bb547eb666046282b130b11a94d933c41537e53849fef: Status 404 returned error can't find the container with id 3b9d2a677fc04337562bb547eb666046282b130b11a94d933c41537e53849fef Oct 30 05:52:43.297112 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:43.297092 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:52:43.318074 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:43.318056 2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-226.ec2.internal\" not found" Oct 30 05:52:43.325611 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:43.325594 2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:52:43.349504 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:43.349488 2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:52:43.408688 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:43.408667 2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:52:43.431730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:43.431684 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" Oct 30 05:52:43.455307 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:43.455290 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:52:43.456413 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:43.456400 2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:52:43.484384 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:43.484366 2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:52:44.107849 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.107811 2575 apiserver.go:52] "Watching apiserver" Oct 30 05:52:44.132602 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.132569 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 05:52:44.154239 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.154179 2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 05:52:44.154598 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.154576 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-dnbrk","kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal","openshift-cluster-node-tuning-operator/tuned-d29qr","openshift-dns/node-resolver-zqrdw","openshift-image-registry/node-ca-vjrj8","openshift-multus/multus-additional-cni-plugins-nbcdv","openshift-multus/network-metrics-daemon-kzpqd","kube-system/global-pull-secret-syncer-pp7mt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal","openshift-multus/multus-zfbpj","kube-system/konnectivity-agent-8mvtx","openshift-network-diagnostics/network-check-target-4gz57","openshift-network-operator/iptables-alerter-xjmhv"] Oct 30 05:52:44.156487 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.156207 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:44.156487 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.156315 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:52:44.157629 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.157603 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:52:44.157725 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.157706 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:52:44.158711 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.158692 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.159975 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.159953 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:52:44.161285 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.161258 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:52:44.162720 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.162702 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.163953 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.163931 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.165914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.165461 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.167165 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.167049 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:52:44.168852 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.168833 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:44.168946 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.168896 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:52:44.170760 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.170743 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.171633 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.171614 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:52:44.181332 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.181209 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 05:52:44.181431 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.181358 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:52:44.181431 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.181425 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q2ncw\"" Oct 30 05:52:44.181543 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.181513 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mpzcb\"" Oct 30 05:52:44.181603 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.181539 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 05:52:44.181603 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.181544 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 05:52:44.181712 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.181614 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 05:52:44.181712 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.181625 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-27cwd\"" Oct 30 05:52:44.181712 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.181677 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 05:52:44.194317 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.194291 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 05:52:44.194423 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.194330 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 05:52:44.194423 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.194391 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 05:52:44.194423 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.194420 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 05:52:44.194585 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.194440 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 05:52:44.195942 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.195924 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 05:52:44.196020 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.196011 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 05:52:44.196162 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.196149 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 05:52:44.196198 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.196185 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 05:52:44.196239 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.196227 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 05:52:44.196313 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.196295 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:52:44.196381 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.196367 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-mkzdc\"" Oct 30 05:52:44.196438 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.196396 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 05:52:44.196438 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.196431 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 05:52:44.196502 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.196400 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d9dv5\"" Oct 30 05:52:44.196502 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.196466 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 05:52:44.208321 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.208303 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 05:52:44.208410 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.208377 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 05:52:44.208467 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.208443 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q46lp\"" Oct 30 05:52:44.208609 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.208592 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5hkvn\"" Oct 30 05:52:44.208670 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.208632 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 05:52:44.208733 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.208671 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 05:52:44.208733 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.208712 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-plsgv\"" Oct 30 05:52:44.208809 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.208779 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 05:52:44.210176 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.210160 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 05:52:44.210248 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.210218 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vzk7w\"" Oct 30 05:52:44.210349 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.210249 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 05:52:44.210412 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.210347 2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 05:52:44.223184 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.223162 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h9mls" Oct 30 05:52:44.232787 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.232772 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 05:52:44.238558 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.238536 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h9mls" Oct 30 05:52:44.245511 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245488 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-kubernetes\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.245664 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245523 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-host\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.245664 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-kubelet\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.245664 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-systemd-units\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.245664 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-cni-binary-copy\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.245836 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245678 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-kubelet\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.245836 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-iptables-alerter-script\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:52:44.245836 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245721 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33bd8834-ad74-4370-b805-a6995dd16230-konnectivity-ca\") pod \"konnectivity-agent-8mvtx\" (UID: \"33bd8834-ad74-4370-b805-a6995dd16230\") " pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:52:44.245836 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245750 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-sys\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.245836 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-system-cni-dir\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.245836 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245809 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-registration-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.245836 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245832 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-sys-fs\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.246117 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-cni-multus\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.246117 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv4cr\" (UniqueName: \"kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr\") pod \"network-check-target-4gz57\" (UID: \"3d693b0d-6752-422d-85d8-e8afb5a111b9\") " pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:52:44.246117 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245897 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-tuned\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.246117 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245913 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cnibin\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.246117 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245932 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-slash\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.246117 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245948 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-cni-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.246117 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.245989 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:44.246117 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246042 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.246117 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246086 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-run-netns\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.246468 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246128 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-log-socket\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.246468 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246177 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-cni-bin\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.246468 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246209 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-socket-dir-parent\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.246468 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246276 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtcl4\" (UniqueName: \"kubernetes.io/projected/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-kube-api-access-gtcl4\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.246468 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246295 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7nmf\" (UniqueName: \"kubernetes.io/projected/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-kube-api-access-b7nmf\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:52:44.246468 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cni-binary-copy\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.246468 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246330 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-daemon-config\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.246468 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246376 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b9d8b00-b8a3-4082-ad89-053a60ff7078-hosts-file\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:52:44.246468 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246411 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpt2s\" (UniqueName: \"kubernetes.io/projected/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-kube-api-access-tpt2s\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:44.246468 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246433 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-run\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.246468 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-etc-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.246468 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246468 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-node-log\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246494 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-cnibin\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-multus-certs\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246592 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysctl-d\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-lib-modules\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-var-lib-kubelet\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246660 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/733fbaf7-e75e-46c7-ad59-478b58424087-host\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246674 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovn-node-metrics-cert\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246723 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovnkube-script-lib\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246748 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-k8s-cni-cncf-io\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246779 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mvkm\" (UniqueName: \"kubernetes.io/projected/12c1aa54-bfc9-4320-915c-59d1701de6d9-kube-api-access-9mvkm\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246803 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-ovn\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246830 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-netns\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246881 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-cni-bin\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.246913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-conf-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.246930 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdl4j\" (UniqueName: \"kubernetes.io/projected/1b9d8b00-b8a3-4082-ad89-053a60ff7078-kube-api-access-zdl4j\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247005 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a968117a-0683-4844-8936-99ef536894d9-dbus\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247033 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-cni-netd\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247061 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovnkube-config\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247083 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-etc-kubernetes\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247111 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-modprobe-d\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhpvd\" (UniqueName: \"kubernetes.io/projected/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-kube-api-access-lhpvd\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247152 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247175 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-system-cni-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247220 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33bd8834-ad74-4370-b805-a6995dd16230-agent-certs\") pod \"konnectivity-agent-8mvtx\" (UID: \"33bd8834-ad74-4370-b805-a6995dd16230\") " pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysctl-conf\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247260 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247287 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-device-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247360 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9d8b00-b8a3-4082-ad89-053a60ff7078-tmp-dir\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:52:44.247614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysconfig\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-systemd\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247452 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-socket-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247467 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-systemd\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-var-lib-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247505 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-host-slash\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247526 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvn5s\" (UniqueName: \"kubernetes.io/projected/733fbaf7-e75e-46c7-ad59-478b58424087-kube-api-access-rvn5s\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247575 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-tmp\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdfl5\" (UniqueName: \"kubernetes.io/projected/dd135f0d-de15-427c-b9f6-490eaf4a7540-kube-api-access-hdfl5\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247677 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-hostroot\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247703 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/733fbaf7-e75e-46c7-ad59-478b58424087-serviceca\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247721 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcqpr\" (UniqueName: \"kubernetes.io/projected/e4a59e77-1b37-405f-b787-df3e7e2ecc43-kube-api-access-dcqpr\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247840 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-etc-selinux\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247862 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-env-overrides\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.248369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.247884 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a968117a-0683-4844-8936-99ef536894d9-kubelet-config\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:44.263693 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.263647 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" event={"ID":"0b8548329cbac7d7fb7409929225464e","Type":"ContainerStarted","Data":"3b9d2a677fc04337562bb547eb666046282b130b11a94d933c41537e53849fef"} Oct 30 05:52:44.265309 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.265285 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" event={"ID":"18e4d93490af52d80822ebfbfe232c01","Type":"ContainerStarted","Data":"8e0e5c1aa27c35c585015e70714593aa7aee54a1ff842315c94d39062be850f9"} Oct 30 05:52:44.348081 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348052 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-socket-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.348183 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-systemd\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.348183 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348114 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-var-lib-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.348183 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348139 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-host-slash\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:52:44.348183 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348164 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvn5s\" (UniqueName: \"kubernetes.io/projected/733fbaf7-e75e-46c7-ad59-478b58424087-kube-api-access-rvn5s\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:52:44.348385 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.348385 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348209 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-tmp\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.348385 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348258 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.348385 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348284 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdfl5\" (UniqueName: \"kubernetes.io/projected/dd135f0d-de15-427c-b9f6-490eaf4a7540-kube-api-access-hdfl5\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.348385 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348280 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-socket-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.348385 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348309 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-hostroot\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.348385 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348332 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/733fbaf7-e75e-46c7-ad59-478b58424087-serviceca\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:52:44.348385 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348355 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcqpr\" (UniqueName: \"kubernetes.io/projected/e4a59e77-1b37-405f-b787-df3e7e2ecc43-kube-api-access-dcqpr\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.348385 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348379 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-etc-selinux\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-env-overrides\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a968117a-0683-4844-8936-99ef536894d9-kubelet-config\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-kubernetes\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348474 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-host\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348498 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-kubelet\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348522 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-systemd-units\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-cni-binary-copy\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348573 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-kubelet\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348579 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-systemd\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348603 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-iptables-alerter-script\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33bd8834-ad74-4370-b805-a6995dd16230-konnectivity-ca\") pod \"konnectivity-agent-8mvtx\" (UID: \"33bd8834-ad74-4370-b805-a6995dd16230\") " pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348636 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-var-lib-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-sys\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348682 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-host-slash\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348682 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-system-cni-dir\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-registration-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.348730 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348726 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-system-cni-dir\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-sys-fs\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348764 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-registration-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348783 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-cni-multus\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348812 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4cr\" (UniqueName: \"kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr\") pod \"network-check-target-4gz57\" (UID: \"3d693b0d-6752-422d-85d8-e8afb5a111b9\") " pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348817 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a968117a-0683-4844-8936-99ef536894d9-kubelet-config\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348839 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-tuned\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348860 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cnibin\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348892 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-slash\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348927 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-cni-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348952 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.348978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.349004 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-run-netns\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.349031 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-log-socket\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.349054 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-cni-bin\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.349079 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-socket-dir-parent\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.349347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.349105 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtcl4\" (UniqueName: \"kubernetes.io/projected/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-kube-api-access-gtcl4\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.350034 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.349131 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7nmf\" (UniqueName: \"kubernetes.io/projected/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-kube-api-access-b7nmf\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:52:44.350034 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.349151 2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 05:52:44.350034 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.349398 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.350034 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.349415 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-kubernetes\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.350034 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.349487 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-host\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.350034 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.349534 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-kubelet\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.350034 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.349574 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-systemd-units\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.350034 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.349684 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-etc-selinux\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.350386 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.350191 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.350386 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.350207 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-env-overrides\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.350386 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.350223 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-cni-binary-copy\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.350386 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.350299 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-kubelet\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.350386 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.350316 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cnibin\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.350588 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.350424 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-cni-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.350588 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.350507 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:44.350588 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.350561 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs podName:f8fac18d-f521-4f23-ad5e-ec2b5dcac722 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:44.85054398 +0000 UTC m=+3.114197376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs") pod "network-metrics-daemon-kzpqd" (UID: "f8fac18d-f521-4f23-ad5e-ec2b5dcac722") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:44.350731 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.350717 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-hostroot\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351050 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-cni-bin\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351104 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-run-netns\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351149 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-log-socket\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351198 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-sys\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351234 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-slash\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351279 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-cni-multus\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cni-binary-copy\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-daemon-config\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b9d8b00-b8a3-4082-ad89-053a60ff7078-hosts-file\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpt2s\" (UniqueName: \"kubernetes.io/projected/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-kube-api-access-tpt2s\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351474 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-run\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351524 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-run\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351525 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b9d8b00-b8a3-4082-ad89-053a60ff7078-hosts-file\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351541 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-etc-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351542 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-socket-dir-parent\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351559 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-iptables-alerter-script\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351579 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-node-log\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.351667 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-node-log\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.352733 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351603 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-etc-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.352733 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-cnibin\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.353310 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353260 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33bd8834-ad74-4370-b805-a6995dd16230-konnectivity-ca\") pod \"konnectivity-agent-8mvtx\" (UID: \"33bd8834-ad74-4370-b805-a6995dd16230\") " pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:52:44.353405 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353335 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-multus-certs\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.353405 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysctl-d\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.353511 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353401 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-sys-fs\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.353511 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353417 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-lib-modules\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.353511 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353455 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-var-lib-kubelet\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.353511 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353491 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/733fbaf7-e75e-46c7-ad59-478b58424087-host\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:52:44.353709 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.353709 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353558 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovn-node-metrics-cert\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.353709 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353609 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovnkube-script-lib\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.353709 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-k8s-cni-cncf-io\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.353709 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353672 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mvkm\" (UniqueName: \"kubernetes.io/projected/12c1aa54-bfc9-4320-915c-59d1701de6d9-kube-api-access-9mvkm\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.353709 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353691 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-ovn\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.354022 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353747 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-ovn\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.354022 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353811 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysctl-d\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.354022 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353879 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-lib-modules\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.354022 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353908 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-var-lib-kubelet\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.354022 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.353945 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/733fbaf7-e75e-46c7-ad59-478b58424087-host\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:52:44.354309 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.351624 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-cnibin\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.354309 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.354155 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-multus-certs\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.354465 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.354441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.354594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.354567 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-daemon-config\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.354749 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.354726 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/733fbaf7-e75e-46c7-ad59-478b58424087-serviceca\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:52:44.356869 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.356842 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.356976 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.356877 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-tuned\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.357044 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.357008 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cni-binary-copy\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.357122 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.357009 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-tmp\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.357953 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.357928 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.357953 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.357931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-netns\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.358125 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.357964 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-k8s-cni-cncf-io\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.358125 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358056 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-cni-bin\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.358125 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358056 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-netns\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.358125 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358100 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-cni-bin\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.358125 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358115 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-conf-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.358428 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdl4j\" (UniqueName: \"kubernetes.io/projected/1b9d8b00-b8a3-4082-ad89-053a60ff7078-kube-api-access-zdl4j\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:52:44.358428 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358185 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a968117a-0683-4844-8936-99ef536894d9-dbus\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:44.358428 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358255 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-cni-netd\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.358428 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovnkube-config\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.358428 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358312 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-etc-kubernetes\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.358428 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358352 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-modprobe-d\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.358428 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhpvd\" (UniqueName: \"kubernetes.io/projected/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-kube-api-access-lhpvd\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.358428 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358403 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.358831 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358424 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovnkube-script-lib\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.358831 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358443 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-system-cni-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.358831 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358470 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:44.358831 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358475 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-conf-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.358831 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358488 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-cni-netd\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.358831 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358519 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-etc-kubernetes\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.358831 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358523 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a968117a-0683-4844-8936-99ef536894d9-dbus\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:44.358831 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358540 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33bd8834-ad74-4370-b805-a6995dd16230-agent-certs\") pod \"konnectivity-agent-8mvtx\" (UID: \"33bd8834-ad74-4370-b805-a6995dd16230\") " pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:52:44.358831 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysctl-conf\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.358831 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.358657 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:44.358831 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358669 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-modprobe-d\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.358831 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.358709 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret podName:a968117a-0683-4844-8936-99ef536894d9 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:44.858696475 +0000 UTC m=+3.122349876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret") pod "global-pull-secret-syncer-pp7mt" (UID: "a968117a-0683-4844-8936-99ef536894d9") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:44.359586 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358927 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.359586 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.358990 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-system-cni-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.359586 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.359029 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovnkube-config\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.359586 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.359103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.359586 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.359125 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-device-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.359586 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.359167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9d8b00-b8a3-4082-ad89-053a60ff7078-tmp-dir\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:52:44.359586 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.359193 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysconfig\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.359586 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.359195 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysctl-conf\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.359586 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.359270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-systemd\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.359586 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.359365 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-systemd\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.359586 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.359432 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysconfig\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.359586 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.359515 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9d8b00-b8a3-4082-ad89-053a60ff7078-tmp-dir\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:52:44.359586 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.359515 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-device-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.360242 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.359798 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.361656 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.361630 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovn-node-metrics-cert\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.361817 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.361781 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33bd8834-ad74-4370-b805-a6995dd16230-agent-certs\") pod \"konnectivity-agent-8mvtx\" (UID: \"33bd8834-ad74-4370-b805-a6995dd16230\") " pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:52:44.380129 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.380064 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:44.380129 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.380091 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:44.380129 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.380104 2575 projected.go:196] Error preparing data for projected volume kube-api-access-tv4cr for pod openshift-network-diagnostics/network-check-target-4gz57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:44.380363 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.380159 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr podName:3d693b0d-6752-422d-85d8-e8afb5a111b9 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:44.880141728 +0000 UTC m=+3.143795115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tv4cr" (UniqueName: "kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr") pod "network-check-target-4gz57" (UID: "3d693b0d-6752-422d-85d8-e8afb5a111b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:44.382724 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.382687 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcqpr\" (UniqueName: \"kubernetes.io/projected/e4a59e77-1b37-405f-b787-df3e7e2ecc43-kube-api-access-dcqpr\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.382833 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.382807 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpt2s\" (UniqueName: \"kubernetes.io/projected/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-kube-api-access-tpt2s\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:44.382833 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.382825 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mvkm\" (UniqueName: \"kubernetes.io/projected/12c1aa54-bfc9-4320-915c-59d1701de6d9-kube-api-access-9mvkm\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.383538 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.383513 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdfl5\" (UniqueName: \"kubernetes.io/projected/dd135f0d-de15-427c-b9f6-490eaf4a7540-kube-api-access-hdfl5\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.383666 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.383645 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtcl4\" (UniqueName: \"kubernetes.io/projected/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-kube-api-access-gtcl4\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.387010 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.386990 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvn5s\" (UniqueName: \"kubernetes.io/projected/733fbaf7-e75e-46c7-ad59-478b58424087-kube-api-access-rvn5s\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:52:44.387733 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.387680 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7nmf\" (UniqueName: \"kubernetes.io/projected/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-kube-api-access-b7nmf\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:52:44.395596 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.395578 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdl4j\" (UniqueName: \"kubernetes.io/projected/1b9d8b00-b8a3-4082-ad89-053a60ff7078-kube-api-access-zdl4j\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:52:44.400605 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.400579 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhpvd\" (UniqueName: \"kubernetes.io/projected/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-kube-api-access-lhpvd\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.474415 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.474331 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:52:44.485196 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:44.485164 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa4aa89_688f_4a17_b2bc_88dbcd856db8.slice/crio-c8ecb23d378885c090ee8391c51aa231b8a89e9446c0495540f8ae0b367290cb WatchSource:0}: Error finding container c8ecb23d378885c090ee8391c51aa231b8a89e9446c0495540f8ae0b367290cb: Status 404 returned error can't find the container with id c8ecb23d378885c090ee8391c51aa231b8a89e9446c0495540f8ae0b367290cb Oct 30 05:52:44.486570 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.486547 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:52:44.494352 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.494323 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:52:44.498151 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:44.498124 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9d8b00_b8a3_4082_ad89_053a60ff7078.slice/crio-f7e2fe7f2d25db83084f7837e3d33968a456f6b72f0ce64a5c252d74b215044f WatchSource:0}: Error finding container f7e2fe7f2d25db83084f7837e3d33968a456f6b72f0ce64a5c252d74b215044f: Status 404 returned error can't find the container with id f7e2fe7f2d25db83084f7837e3d33968a456f6b72f0ce64a5c252d74b215044f Oct 30 05:52:44.504688 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:44.504464 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733fbaf7_e75e_46c7_ad59_478b58424087.slice/crio-da76d95f71989097ae5d4acee3f2d4b25c42927e0b4446c5ca5fb951e07ffef5 WatchSource:0}: Error finding container da76d95f71989097ae5d4acee3f2d4b25c42927e0b4446c5ca5fb951e07ffef5: Status 404 returned error can't find the container with id da76d95f71989097ae5d4acee3f2d4b25c42927e0b4446c5ca5fb951e07ffef5 Oct 30 05:52:44.510553 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.510534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:52:44.518133 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.518110 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:52:44.519532 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:44.519497 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a59e77_1b37_405f_b787_df3e7e2ecc43.slice/crio-ce7779c1faa006e750c121aab96fca0ab9780f2addad1ebbb5767d03f962bb98 WatchSource:0}: Error finding container ce7779c1faa006e750c121aab96fca0ab9780f2addad1ebbb5767d03f962bb98: Status 404 returned error can't find the container with id ce7779c1faa006e750c121aab96fca0ab9780f2addad1ebbb5767d03f962bb98 Oct 30 05:52:44.526423 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:44.526388 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd135f0d_de15_427c_b9f6_490eaf4a7540.slice/crio-42fec8443c5340c6d46575e7b9429eda65c9d6f375cb53008cfc97aad156c140 WatchSource:0}: Error finding container 42fec8443c5340c6d46575e7b9429eda65c9d6f375cb53008cfc97aad156c140: Status 404 returned error can't find the container with id 42fec8443c5340c6d46575e7b9429eda65c9d6f375cb53008cfc97aad156c140 Oct 30 05:52:44.527155 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.527136 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:52:44.528278 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.528260 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:52:44.536536 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.536514 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zfbpj" Oct 30 05:52:44.542024 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:44.542001 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c1aa54_bfc9_4320_915c_59d1701de6d9.slice/crio-fb64e44a31c75bf1d9c5032529d5a1ac94d9438b1438f51e622a61ec7653921c WatchSource:0}: Error finding container fb64e44a31c75bf1d9c5032529d5a1ac94d9438b1438f51e622a61ec7653921c: Status 404 returned error can't find the container with id fb64e44a31c75bf1d9c5032529d5a1ac94d9438b1438f51e622a61ec7653921c Oct 30 05:52:44.542828 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.542808 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:52:44.545739 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:44.545718 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ae8ad0_aec2_4513_a5b5_75e6246d92fb.slice/crio-edcd238349cc00cdebb507db0ad6a83da6bceefa05545386605df62f8634b5ce WatchSource:0}: Error finding container edcd238349cc00cdebb507db0ad6a83da6bceefa05545386605df62f8634b5ce: Status 404 returned error can't find the container with id edcd238349cc00cdebb507db0ad6a83da6bceefa05545386605df62f8634b5ce Oct 30 05:52:44.550107 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:44.550083 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c62aa1_d87b_4997_ae6a_ed1957b6ea73.slice/crio-762f4b706be830b590590fef507451cabf71f5ec49bd6e9fde27f3cf55103487 WatchSource:0}: Error finding container 762f4b706be830b590590fef507451cabf71f5ec49bd6e9fde27f3cf55103487: Status 404 returned error can't find the container with id 762f4b706be830b590590fef507451cabf71f5ec49bd6e9fde27f3cf55103487 Oct 30 05:52:44.553956 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:52:44.553934 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33bd8834_ad74_4370_b805_a6995dd16230.slice/crio-f01bd5ac744e47ae83a3846db47ffed5b89b9a99c5411662a7fc4a55248dc056 WatchSource:0}: Error finding container f01bd5ac744e47ae83a3846db47ffed5b89b9a99c5411662a7fc4a55248dc056: Status 404 returned error can't find the container with id f01bd5ac744e47ae83a3846db47ffed5b89b9a99c5411662a7fc4a55248dc056 Oct 30 05:52:44.682140 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.682109 2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:52:44.863136 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.863104 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:44.863262 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.863181 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:44.863370 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.863350 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:44.863474 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.863416 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret podName:a968117a-0683-4844-8936-99ef536894d9 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:45.863397045 +0000 UTC m=+4.127050420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret") pod "global-pull-secret-syncer-pp7mt" (UID: "a968117a-0683-4844-8936-99ef536894d9") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:44.863474 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.863442 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:44.863570 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.863500 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs podName:f8fac18d-f521-4f23-ad5e-ec2b5dcac722 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:45.863486555 +0000 UTC m=+4.127139927 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs") pod "network-metrics-daemon-kzpqd" (UID: "f8fac18d-f521-4f23-ad5e-ec2b5dcac722") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:44.964424 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:44.964314 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4cr\" (UniqueName: \"kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr\") pod \"network-check-target-4gz57\" (UID: \"3d693b0d-6752-422d-85d8-e8afb5a111b9\") " pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:52:44.964551 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.964523 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:44.964613 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.964554 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:44.964613 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.964568 2575 projected.go:196] Error preparing data for projected volume kube-api-access-tv4cr for pod openshift-network-diagnostics/network-check-target-4gz57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:44.964707 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:44.964630 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr podName:3d693b0d-6752-422d-85d8-e8afb5a111b9 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:45.964608657 +0000 UTC m=+4.228262043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tv4cr" (UniqueName: "kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr") pod "network-check-target-4gz57" (UID: "3d693b0d-6752-422d-85d8-e8afb5a111b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:45.240799 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.240675 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:47:45 +0000 UTC" deadline="2028-06-16 14:32:10.294449584 +0000 UTC" Oct 30 05:52:45.240799 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.240746 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14288h39m25.053707733s" Oct 30 05:52:45.260647 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.258840 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:45.260647 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:45.258962 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:52:45.271478 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.271405 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8mvtx" event={"ID":"33bd8834-ad74-4370-b805-a6995dd16230","Type":"ContainerStarted","Data":"f01bd5ac744e47ae83a3846db47ffed5b89b9a99c5411662a7fc4a55248dc056"} Oct 30 05:52:45.274491 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.274422 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbcdv" event={"ID":"e4a59e77-1b37-405f-b787-df3e7e2ecc43","Type":"ContainerStarted","Data":"ce7779c1faa006e750c121aab96fca0ab9780f2addad1ebbb5767d03f962bb98"} Oct 30 05:52:45.278594 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.278377 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zqrdw" event={"ID":"1b9d8b00-b8a3-4082-ad89-053a60ff7078","Type":"ContainerStarted","Data":"f7e2fe7f2d25db83084f7837e3d33968a456f6b72f0ce64a5c252d74b215044f"} Oct 30 05:52:45.283090 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.282231 2575 generic.go:356] "Generic (PLEG): container finished" podID="18e4d93490af52d80822ebfbfe232c01" containerID="4cb029c4d74de9c2c479a9de9b40cc3b203c48fb8c5bcb0ec406649f82d4ddb7" exitCode=0 Oct 30 05:52:45.283329 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.283259 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" event={"ID":"18e4d93490af52d80822ebfbfe232c01","Type":"ContainerDied","Data":"4cb029c4d74de9c2c479a9de9b40cc3b203c48fb8c5bcb0ec406649f82d4ddb7"} Oct 30 05:52:45.293808 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.293758 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zfbpj" event={"ID":"26c62aa1-d87b-4997-ae6a-ed1957b6ea73","Type":"ContainerStarted","Data":"762f4b706be830b590590fef507451cabf71f5ec49bd6e9fde27f3cf55103487"} Oct 30 05:52:45.297947 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.297875 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xjmhv" event={"ID":"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb","Type":"ContainerStarted","Data":"edcd238349cc00cdebb507db0ad6a83da6bceefa05545386605df62f8634b5ce"} Oct 30 05:52:45.302706 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.302577 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" event={"ID":"12c1aa54-bfc9-4320-915c-59d1701de6d9","Type":"ContainerStarted","Data":"fb64e44a31c75bf1d9c5032529d5a1ac94d9438b1438f51e622a61ec7653921c"} Oct 30 05:52:45.305955 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.305926 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" event={"ID":"dd135f0d-de15-427c-b9f6-490eaf4a7540","Type":"ContainerStarted","Data":"42fec8443c5340c6d46575e7b9429eda65c9d6f375cb53008cfc97aad156c140"} Oct 30 05:52:45.309002 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.308974 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vjrj8" event={"ID":"733fbaf7-e75e-46c7-ad59-478b58424087","Type":"ContainerStarted","Data":"da76d95f71989097ae5d4acee3f2d4b25c42927e0b4446c5ca5fb951e07ffef5"} Oct 30 05:52:45.310841 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.310816 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d29qr" event={"ID":"6aa4aa89-688f-4a17-b2bc-88dbcd856db8","Type":"ContainerStarted","Data":"c8ecb23d378885c090ee8391c51aa231b8a89e9446c0495540f8ae0b367290cb"} Oct 30 05:52:45.315975 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.315948 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" event={"ID":"0b8548329cbac7d7fb7409929225464e","Type":"ContainerStarted","Data":"fd40ada5509e946891ea0e4f3dbcc8c07304aa2ec77ff0690f9e5cb7235a8901"} Oct 30 05:52:45.872881 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.872578 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:45.872881 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.872873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:45.873115 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:45.872885 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:45.873115 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:45.872966 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs podName:f8fac18d-f521-4f23-ad5e-ec2b5dcac722 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:47.872944881 +0000 UTC m=+6.136598257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs") pod "network-metrics-daemon-kzpqd" (UID: "f8fac18d-f521-4f23-ad5e-ec2b5dcac722") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:45.873115 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:45.872987 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:45.873115 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:45.873040 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret podName:a968117a-0683-4844-8936-99ef536894d9 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:47.873024698 +0000 UTC m=+6.136678082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret") pod "global-pull-secret-syncer-pp7mt" (UID: "a968117a-0683-4844-8936-99ef536894d9") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:45.973622 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:45.973589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4cr\" (UniqueName: \"kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr\") pod \"network-check-target-4gz57\" (UID: \"3d693b0d-6752-422d-85d8-e8afb5a111b9\") " pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:52:45.973772 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:45.973734 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:45.973772 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:45.973751 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:45.973772 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:45.973761 2575 projected.go:196] Error preparing data for projected volume kube-api-access-tv4cr for pod openshift-network-diagnostics/network-check-target-4gz57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:45.973914 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:45.973800 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr podName:3d693b0d-6752-422d-85d8-e8afb5a111b9 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:47.973788431 +0000 UTC m=+6.237441814 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tv4cr" (UniqueName: "kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr") pod "network-check-target-4gz57" (UID: "3d693b0d-6752-422d-85d8-e8afb5a111b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:46.242963 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:46.242889 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:47:45 +0000 UTC" deadline="2028-03-30 22:34:14.790431886 +0000 UTC" Oct 30 05:52:46.242963 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:46.242928 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12424h41m28.54750819s" Oct 30 05:52:46.261052 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:46.261024 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:46.261224 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:46.261198 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:52:46.261327 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:46.261308 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:52:46.261445 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:46.261392 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:52:46.337379 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:46.337344 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" event={"ID":"18e4d93490af52d80822ebfbfe232c01","Type":"ContainerStarted","Data":"d2e85163fecdb49037614b5043abb3f73fc01dd68863dd945feaa0ccea8fd22d"} Oct 30 05:52:46.384793 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:46.384156 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" podStartSLOduration=2.384137506 podStartE2EDuration="2.384137506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:45.371267426 +0000 UTC m=+3.634920820" watchObservedRunningTime="2026-10-30 05:52:46.384137506 +0000 UTC m=+4.647790901" Oct 30 05:52:47.259432 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:47.258937 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:47.259432 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:47.259077 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:52:47.890559 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:47.890478 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:47.890559 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:47.890541 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:47.890777 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:47.890676 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:47.890777 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:47.890739 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret podName:a968117a-0683-4844-8936-99ef536894d9 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:51.890718762 +0000 UTC m=+10.154372149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret") pod "global-pull-secret-syncer-pp7mt" (UID: "a968117a-0683-4844-8936-99ef536894d9") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:47.891166 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:47.891140 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:47.891284 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:47.891203 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs podName:f8fac18d-f521-4f23-ad5e-ec2b5dcac722 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:51.89118439 +0000 UTC m=+10.154837774 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs") pod "network-metrics-daemon-kzpqd" (UID: "f8fac18d-f521-4f23-ad5e-ec2b5dcac722") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:47.991420 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:47.991390 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4cr\" (UniqueName: \"kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr\") pod \"network-check-target-4gz57\" (UID: \"3d693b0d-6752-422d-85d8-e8afb5a111b9\") " pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:52:47.991600 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:47.991533 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:47.991600 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:47.991551 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:47.991600 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:47.991562 2575 projected.go:196] Error preparing data for projected volume kube-api-access-tv4cr for pod openshift-network-diagnostics/network-check-target-4gz57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:47.991771 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:47.991621 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr podName:3d693b0d-6752-422d-85d8-e8afb5a111b9 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:51.991603077 +0000 UTC m=+10.255256463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tv4cr" (UniqueName: "kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr") pod "network-check-target-4gz57" (UID: "3d693b0d-6752-422d-85d8-e8afb5a111b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:48.264861 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:48.264787 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:48.265336 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:48.264929 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:52:48.265408 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:48.265369 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:52:48.265494 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:48.265470 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:52:48.353061 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:48.353005 2575 generic.go:356] "Generic (PLEG): container finished" podID="e4a59e77-1b37-405f-b787-df3e7e2ecc43" containerID="4b40b3fdac93e95a3ca78718321d91f60b1308e252c43a8e8248c67e5f471db1" exitCode=0 Oct 30 05:52:48.353061 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:48.353064 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbcdv" event={"ID":"e4a59e77-1b37-405f-b787-df3e7e2ecc43","Type":"ContainerDied","Data":"4b40b3fdac93e95a3ca78718321d91f60b1308e252c43a8e8248c67e5f471db1"} Oct 30 05:52:48.436414 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:48.436350 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" podStartSLOduration=4.436330625 podStartE2EDuration="4.436330625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:46.384775587 +0000 UTC m=+4.648428981" watchObservedRunningTime="2026-10-30 05:52:48.436330625 +0000 UTC m=+6.699984022" Oct 30 05:52:49.259728 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:49.259186 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:49.259728 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:49.259348 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:52:49.359672 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:49.358947 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8mvtx" event={"ID":"33bd8834-ad74-4370-b805-a6995dd16230","Type":"ContainerStarted","Data":"76067c9c9a541379a3e99a71398846437c62f1fd797c989e1a05155ad08d96f0"} Oct 30 05:52:50.258760 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:50.258730 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:52:50.258915 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:50.258862 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:52:50.259339 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:50.259285 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:50.259422 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:50.259399 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:52:50.365098 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:50.364998 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vjrj8" event={"ID":"733fbaf7-e75e-46c7-ad59-478b58424087","Type":"ContainerStarted","Data":"7826c59ada7c4bc6695fbb679282e46231be967641065408e57e4341bbbf8066"} Oct 30 05:52:50.412060 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:50.411959 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8mvtx" podStartSLOduration=3.450650102 podStartE2EDuration="7.411938322s" totalImagesPullingTime="3.96128822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:49.406535544 +0000 UTC m=+7.670188938" watchObservedRunningTime="2026-10-30 05:52:50.411938322 +0000 UTC m=+8.675591718" Oct 30 05:52:51.259265 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:51.259205 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:51.259460 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:51.259378 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:52:51.372066 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:51.372025 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" event={"ID":"dd135f0d-de15-427c-b9f6-490eaf4a7540","Type":"ContainerStarted","Data":"163f94fdae4d6db1b3a183cf0cbda46aa28e5f214b90ff4d6b38264bad4aed60"} Oct 30 05:52:51.926856 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:51.926816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:51.927056 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:51.926895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:51.927056 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:51.927046 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:51.927176 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:51.927113 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret podName:a968117a-0683-4844-8936-99ef536894d9 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:59.927092529 +0000 UTC m=+18.190745900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret") pod "global-pull-secret-syncer-pp7mt" (UID: "a968117a-0683-4844-8936-99ef536894d9") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:51.927579 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:51.927557 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:51.927670 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:51.927625 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs podName:f8fac18d-f521-4f23-ad5e-ec2b5dcac722 nodeName:}" failed. No retries permitted until 2026-10-30 05:52:59.927606608 +0000 UTC m=+18.191259979 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs") pod "network-metrics-daemon-kzpqd" (UID: "f8fac18d-f521-4f23-ad5e-ec2b5dcac722") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:52.028338 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:52.028299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4cr\" (UniqueName: \"kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr\") pod \"network-check-target-4gz57\" (UID: \"3d693b0d-6752-422d-85d8-e8afb5a111b9\") " pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:52:52.028507 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:52.028485 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:52:52.028550 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:52.028509 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:52:52.028550 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:52.028528 2575 projected.go:196] Error preparing data for projected volume kube-api-access-tv4cr for pod openshift-network-diagnostics/network-check-target-4gz57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:52.028618 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:52.028582 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr podName:3d693b0d-6752-422d-85d8-e8afb5a111b9 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:00.028568093 +0000 UTC m=+18.292221461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tv4cr" (UniqueName: "kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr") pod "network-check-target-4gz57" (UID: "3d693b0d-6752-422d-85d8-e8afb5a111b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:52:52.263958 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:52.263481 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:52:52.263958 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:52.263516 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:52.263958 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:52.263597 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:52:52.263958 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:52.263654 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:52:52.340938 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:52.340906 2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 05:52:52.376532 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:52.376501 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" event={"ID":"dd135f0d-de15-427c-b9f6-490eaf4a7540","Type":"ContainerStarted","Data":"a77228ef0291dbed46f0168f39a064f0169b8ff2973976bf471bec81171e79ec"} Oct 30 05:52:52.566507 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:52.566452 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:52:52.566783 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:52.566762 2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:52:52.613311 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:52.613212 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vjrj8" podStartSLOduration=4.597510666 podStartE2EDuration="9.613195144s" totalImagesPullingTime="5.015684478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:50.412692807 +0000 UTC m=+8.676346202" watchObservedRunningTime="2026-10-30 05:52:52.613195144 +0000 UTC m=+10.876848539" Oct 30 05:52:53.196657 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:53.195512 2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:52:52.340928942Z","UUID":"56b3b68f-c3d6-4744-959d-380c5108b93a","Handler":null,"Name":"","Endpoint":""} Oct 30 05:52:53.198239 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:53.197765 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 05:52:53.198239 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:53.197796 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 05:52:53.259343 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:53.259303 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:53.259556 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:53.259430 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:52:53.381749 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:53.381673 2575 generic.go:356] "Generic (PLEG): container finished" podID="e4a59e77-1b37-405f-b787-df3e7e2ecc43" containerID="643547053d5eba4a4debe8d85639c56b0446fd4ceeb430bbbf5855276a708613" exitCode=0 Oct 30 05:52:53.382225 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:53.382007 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbcdv" event={"ID":"e4a59e77-1b37-405f-b787-df3e7e2ecc43","Type":"ContainerDied","Data":"643547053d5eba4a4debe8d85639c56b0446fd4ceeb430bbbf5855276a708613"} Oct 30 05:52:53.388638 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:53.388608 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zqrdw" event={"ID":"1b9d8b00-b8a3-4082-ad89-053a60ff7078","Type":"ContainerStarted","Data":"4bd5e8adda31464f1923ec4ce9447ad180a14200de6e0f398a0db3174dba169f"} Oct 30 05:52:53.391214 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:53.391194 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:52:53.391606 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:53.391584 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zfbpj" event={"ID":"26c62aa1-d87b-4997-ae6a-ed1957b6ea73","Type":"ContainerStarted","Data":"7984e0906cfbb5c50b52c0ff851bfc66eff61dd91dc0419ff59a399f59b55d15"} Oct 30 05:52:53.540000 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:53.539930 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-zfbpj" podStartSLOduration=2.516629882 podStartE2EDuration="10.53991001s" totalImagesPullingTime="8.023280128s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:53.489334992 +0000 UTC m=+11.752988386" watchObservedRunningTime="2026-10-30 05:52:53.53991001 +0000 UTC m=+11.803563403" Oct 30 05:52:54.259382 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:54.259121 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:54.264524 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:54.264479 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:52:54.265111 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:54.265091 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:52:54.265272 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:54.265251 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:52:54.395313 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:54.395205 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xjmhv" event={"ID":"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb","Type":"ContainerStarted","Data":"dc17180672c410d1d608c10c9c439370e0d409e2aaaef8ba688f0e5544bc27df"} Oct 30 05:52:54.399516 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:54.399481 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" event={"ID":"dd135f0d-de15-427c-b9f6-490eaf4a7540","Type":"ContainerStarted","Data":"8691cfa6aa6d4a395fabbd687fed731427b531f2fde8ae79a6b336435a31f909"} Oct 30 05:52:54.402479 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:54.402452 2575 generic.go:356] "Generic (PLEG): container finished" podID="e4a59e77-1b37-405f-b787-df3e7e2ecc43" containerID="6e27ea621e3ee4bb978efed557cd158630876f808b7cd793e1df368774628b27" exitCode=0 Oct 30 05:52:54.402568 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:54.402549 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbcdv" event={"ID":"e4a59e77-1b37-405f-b787-df3e7e2ecc43","Type":"ContainerDied","Data":"6e27ea621e3ee4bb978efed557cd158630876f808b7cd793e1df368774628b27"} Oct 30 05:52:54.439647 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:54.439594 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xjmhv" podStartSLOduration=3.556849113 podStartE2EDuration="11.439577395s" totalImagesPullingTime="7.882728282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:54.438950376 +0000 UTC m=+12.702603799" watchObservedRunningTime="2026-10-30 05:52:54.439577395 +0000 UTC m=+12.703230790" Oct 30 05:52:54.440064 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:54.440020 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-zqrdw" podStartSLOduration=3.116315672 podStartE2EDuration="11.440004667s" totalImagesPullingTime="8.323688995s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:53.540984898 +0000 UTC m=+11.804638290" watchObservedRunningTime="2026-10-30 05:52:54.440004667 +0000 UTC m=+12.703658061" Oct 30 05:52:54.998471 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:54.998443 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:52:55.253542 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:55.253518 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:52:55.258198 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:55.258174 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:55.258365 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:55.258341 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:52:55.407591 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:55.407554 2575 generic.go:356] "Generic (PLEG): container finished" podID="e4a59e77-1b37-405f-b787-df3e7e2ecc43" containerID="3269e40fbd2d6a16e95e5eda27a89e91b36456e35d61eed5f3acea181f7a65ab" exitCode=0 Oct 30 05:52:55.408229 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:55.407655 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbcdv" event={"ID":"e4a59e77-1b37-405f-b787-df3e7e2ecc43","Type":"ContainerDied","Data":"3269e40fbd2d6a16e95e5eda27a89e91b36456e35d61eed5f3acea181f7a65ab"} Oct 30 05:52:55.409918 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:55.409737 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-d29qr" event={"ID":"6aa4aa89-688f-4a17-b2bc-88dbcd856db8","Type":"ContainerStarted","Data":"c93afba6286a9bd5cb675080b888a288dc44bc4aa89f732e662b70fafe32c657"} Oct 30 05:52:55.471379 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:55.471326 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" podStartSLOduration=3.985134891 podStartE2EDuration="12.471312217s" totalImagesPullingTime="8.486177326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:54.543085935 +0000 UTC m=+12.806739332" watchObservedRunningTime="2026-10-30 05:52:55.471312217 +0000 UTC m=+13.734965609" Oct 30 05:52:55.534085 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:55.533996 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-d29qr" podStartSLOduration=2.026497331 podStartE2EDuration="12.533978612s" totalImagesPullingTime="10.507481281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:55.532941031 +0000 UTC m=+13.796594424" watchObservedRunningTime="2026-10-30 05:52:55.533978612 +0000 UTC m=+13.797632006" Oct 30 05:52:56.260399 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:56.260362 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:52:56.260582 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:56.260485 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:52:56.260828 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:56.260809 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:56.260943 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:56.260925 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:52:57.258468 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:57.258444 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:57.259066 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:57.258547 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:52:57.418225 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:57.417974 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbcdv" event={"ID":"e4a59e77-1b37-405f-b787-df3e7e2ecc43","Type":"ContainerStarted","Data":"b96b894fcbc47bab7ffe04004e4199cf75f255255ad282f4240d2c147c4bb312"} Oct 30 05:52:58.258516 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:58.258482 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:58.258928 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:58.258620 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:52:58.258928 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:58.258680 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:52:58.258928 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:58.258793 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:52:58.422548 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:58.422516 2575 generic.go:356] "Generic (PLEG): container finished" podID="e4a59e77-1b37-405f-b787-df3e7e2ecc43" containerID="b96b894fcbc47bab7ffe04004e4199cf75f255255ad282f4240d2c147c4bb312" exitCode=0 Oct 30 05:52:58.422715 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:58.422577 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbcdv" event={"ID":"e4a59e77-1b37-405f-b787-df3e7e2ecc43","Type":"ContainerDied","Data":"b96b894fcbc47bab7ffe04004e4199cf75f255255ad282f4240d2c147c4bb312"} Oct 30 05:52:59.072688 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:59.072651 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:52:59.213121 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:59.213093 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:52:59.258711 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:59.258685 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:59.259036 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:59.258786 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:52:59.427516 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:59.427453 2575 generic.go:356] "Generic (PLEG): container finished" podID="e4a59e77-1b37-405f-b787-df3e7e2ecc43" containerID="c14da276786db7ef8e7f0d631410f083b423eff2d6fd0b5c4236056114253514" exitCode=0 Oct 30 05:52:59.427516 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:59.427493 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbcdv" event={"ID":"e4a59e77-1b37-405f-b787-df3e7e2ecc43","Type":"ContainerDied","Data":"c14da276786db7ef8e7f0d631410f083b423eff2d6fd0b5c4236056114253514"} Oct 30 05:52:59.994579 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:59.994557 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:52:59.994661 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:52:59.994607 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:52:59.994724 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:59.994691 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:59.994724 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:59.994702 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:52:59.994796 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:59.994742 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs podName:f8fac18d-f521-4f23-ad5e-ec2b5dcac722 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:15.994728724 +0000 UTC m=+34.258382107 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs") pod "network-metrics-daemon-kzpqd" (UID: "f8fac18d-f521-4f23-ad5e-ec2b5dcac722") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:52:59.994796 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:52:59.994760 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret podName:a968117a-0683-4844-8936-99ef536894d9 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:15.994750377 +0000 UTC m=+34.258403751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret") pod "global-pull-secret-syncer-pp7mt" (UID: "a968117a-0683-4844-8936-99ef536894d9") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:53:00.095012 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:00.094990 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4cr\" (UniqueName: \"kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr\") pod \"network-check-target-4gz57\" (UID: \"3d693b0d-6752-422d-85d8-e8afb5a111b9\") " pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:00.095237 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:00.095146 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:53:00.095237 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:00.095166 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:53:00.095237 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:00.095178 2575 projected.go:196] Error preparing data for projected volume kube-api-access-tv4cr for pod openshift-network-diagnostics/network-check-target-4gz57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:53:00.095351 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:00.095252 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr podName:3d693b0d-6752-422d-85d8-e8afb5a111b9 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:16.095234683 +0000 UTC m=+34.358888064 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tv4cr" (UniqueName: "kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr") pod "network-check-target-4gz57" (UID: "3d693b0d-6752-422d-85d8-e8afb5a111b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:53:00.258756 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:00.258731 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:00.259087 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:00.258863 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:00.259087 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:00.258895 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:00.259087 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:00.258951 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:00.432281 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:00.432093 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" event={"ID":"12c1aa54-bfc9-4320-915c-59d1701de6d9","Type":"ContainerStarted","Data":"deec456f18e1e9b9674afb759a586afc0c5d02b69f48f0891be6f3d94a6a449c"} Oct 30 05:53:00.432429 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:00.432324 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" event={"ID":"12c1aa54-bfc9-4320-915c-59d1701de6d9","Type":"ContainerStarted","Data":"9b1cde37fe58b93ad66c062e5f60c5f554a39459d479235d132c704e2ce079d6"} Oct 30 05:53:00.432429 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:00.432355 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" event={"ID":"12c1aa54-bfc9-4320-915c-59d1701de6d9","Type":"ContainerStarted","Data":"9224cec325d382b9dfa1d5154c6248d56dc847f63ddaa5c1c81f89b5748a1d14"} Oct 30 05:53:00.432429 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:00.432392 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" event={"ID":"12c1aa54-bfc9-4320-915c-59d1701de6d9","Type":"ContainerStarted","Data":"9778b963b0acc92798fdcc3178b66b620c92cb67c697108ba14ea1c237b1cd6a"} Oct 30 05:53:00.432429 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:00.432423 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" event={"ID":"12c1aa54-bfc9-4320-915c-59d1701de6d9","Type":"ContainerStarted","Data":"b7a9b59122caae1cc3bdfdac6ef52a9fb6ae5daa7417c8200167e04c5d8fbbd0"} Oct 30 05:53:00.432580 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:00.432463 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" event={"ID":"12c1aa54-bfc9-4320-915c-59d1701de6d9","Type":"ContainerStarted","Data":"f613ead6bb590a689ac96cfd308c084a5f8697b1316ffb0ef4ca2a553cacc507"} Oct 30 05:53:00.435125 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:00.435093 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nbcdv" event={"ID":"e4a59e77-1b37-405f-b787-df3e7e2ecc43","Type":"ContainerStarted","Data":"cd377e078269a9a87b046cc2f6ce9ed62b468abbb548970ff11cf68e0df24e46"} Oct 30 05:53:00.486965 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:00.486903 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nbcdv" podStartSLOduration=6.838860099 podStartE2EDuration="17.486891296s" totalImagesPullingTime="10.648031197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:00.486853753 +0000 UTC m=+18.750507141" watchObservedRunningTime="2026-10-30 05:53:00.486891296 +0000 UTC m=+18.750544685" Oct 30 05:53:00.765287 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:00.765207 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:00.948022 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:00.947996 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:01.258490 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:01.258421 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:01.258618 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:01.258523 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:53:01.614674 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:01.614602 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:01.806306 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:01.806272 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:01.965768 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:01.965702 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:02.139177 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:02.139154 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:02.261418 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:02.261344 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:02.261557 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:02.261503 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:02.261557 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:02.261528 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:02.261678 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:02.261594 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:02.291908 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:02.291884 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:53:02.292031 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:02.292017 2575 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:53:02.292442 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:02.292424 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:53:02.443039 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:02.443006 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" event={"ID":"12c1aa54-bfc9-4320-915c-59d1701de6d9","Type":"ContainerStarted","Data":"c015eb8d1585be2e2e568fde4b56a54a006831d8ae5fa8f486a08fa1aedf945c"} Oct 30 05:53:02.545229 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:02.545143 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:02.890464 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:02.890390 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:03.039540 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:03.039508 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:03.258778 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:03.258707 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:03.258922 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:03.258804 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:53:03.290961 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:03.290934 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:03.808863 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:03.808831 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:03.957493 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:03.957465 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:04.258879 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:04.258797 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:04.259023 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:04.258919 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:04.259023 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:04.258991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:04.259118 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:04.259064 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:04.310080 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:04.310048 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:04.454045 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:04.451079 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" event={"ID":"12c1aa54-bfc9-4320-915c-59d1701de6d9","Type":"ContainerStarted","Data":"de05d4adcb15825301a682394db72f80baebf5064e019c04615264335f9d252c"} Oct 30 05:53:04.454045 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:04.453164 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:04.454045 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:04.453394 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:04.454045 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:04.453475 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:04.470192 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:04.470165 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:04.470373 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:04.470301 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:04.548172 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:04.548079 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" podStartSLOduration=6.164382177 podStartE2EDuration="21.548065929s" totalImagesPullingTime="15.383683752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:04.535007758 +0000 UTC m=+22.798661146" watchObservedRunningTime="2026-10-30 05:53:04.548065929 +0000 UTC m=+22.811719318" Oct 30 05:53:05.258670 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:05.258646 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:05.259402 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:05.258741 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:53:05.375521 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:05.375490 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zqrdw_1b9d8b00-b8a3-4082-ad89-053a60ff7078/dns-node-resolver/0.log" Oct 30 05:53:06.263192 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:06.263164 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:06.263610 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:06.263317 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:06.263610 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:06.263359 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:06.263610 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:06.263434 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:06.376052 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:06.376028 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vjrj8_733fbaf7-e75e-46c7-ad59-478b58424087/node-ca/0.log" Oct 30 05:53:06.539294 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:06.539225 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:06.963445 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:06.963365 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:07.182360 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:07.182333 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:07.258369 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:07.258312 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:07.258479 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:07.258403 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:53:07.361383 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:07.361356 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:07.729273 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:07.729191 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:08.047887 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:08.047819 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:08.261155 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:08.261133 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:08.261308 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:08.261137 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:08.261308 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:08.261227 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:08.261386 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:08.261308 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:08.405248 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:08.405171 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:08.558790 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:08.558754 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:08.698804 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:08.698738 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:08.847519 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:08.847495 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:08.992009 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:08.991949 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:09.140337 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:09.140305 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:09.258459 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:09.258410 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:09.258542 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:09.258493 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:53:09.316102 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:09.316080 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:09.524130 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:09.524073 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:10.259191 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:10.259161 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:10.259383 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:10.259281 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:10.259383 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:10.259355 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:10.259499 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:10.259436 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:10.651963 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:10.651890 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:10.814591 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:10.814561 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:10.992184 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:10.992096 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:11.158098 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:11.158073 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:11.258587 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:11.258524 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:11.258718 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:11.258630 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:53:11.332282 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:11.332246 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:11.517658 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:11.517584 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:11.970828 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:11.970757 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:12.110699 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:12.110674 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:12.259910 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:12.259838 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:12.260052 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:12.260002 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:12.260117 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:12.260055 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:12.260169 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:12.260155 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:12.725311 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:12.725279 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:12.932162 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:12.932135 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:13.123025 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:13.122996 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:13.258203 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:13.258174 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:13.258363 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:13.258287 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:53:13.320692 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:13.320661 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:13.528194 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:13.528112 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:14.258873 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:14.258842 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:14.259347 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:14.258957 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:14.259347 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:14.259025 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:14.259347 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:14.259119 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:14.615608 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:14.615580 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:14.820965 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:14.820796 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:14.960532 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:14.960449 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:15.125652 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:15.125620 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:15.258606 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:15.258548 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:15.258760 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:15.258651 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:53:15.327815 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:15.327794 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:15.523109 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:15.523045 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:15.694088 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:15.694063 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:15.848154 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:15.848098 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:16.010506 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:16.010477 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:16.010671 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:16.010529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:16.010671 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:16.010613 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:53:16.010671 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:16.010643 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:53:16.010942 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:16.010921 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret podName:a968117a-0683-4844-8936-99ef536894d9 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:48.010896762 +0000 UTC m=+66.274550142 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret") pod "global-pull-secret-syncer-pp7mt" (UID: "a968117a-0683-4844-8936-99ef536894d9") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:53:16.011126 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:16.011114 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs podName:f8fac18d-f521-4f23-ad5e-ec2b5dcac722 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:48.011077479 +0000 UTC m=+66.274730846 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs") pod "network-metrics-daemon-kzpqd" (UID: "f8fac18d-f521-4f23-ad5e-ec2b5dcac722") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:53:16.111665 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:16.111646 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4cr\" (UniqueName: \"kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr\") pod \"network-check-target-4gz57\" (UID: \"3d693b0d-6752-422d-85d8-e8afb5a111b9\") " pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:16.111780 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:16.111748 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:53:16.111780 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:16.111761 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:53:16.111780 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:16.111770 2575 projected.go:196] Error preparing data for projected volume kube-api-access-tv4cr for pod openshift-network-diagnostics/network-check-target-4gz57: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:53:16.111894 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:16.111808 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr podName:3d693b0d-6752-422d-85d8-e8afb5a111b9 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:48.111798133 +0000 UTC m=+66.375451516 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tv4cr" (UniqueName: "kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr") pod "network-check-target-4gz57" (UID: "3d693b0d-6752-422d-85d8-e8afb5a111b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:53:16.258943 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:16.258917 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:16.259069 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:16.258920 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:16.259069 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:16.259020 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:16.259171 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:16.259070 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:16.737955 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:16.737923 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:16.916474 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:16.916442 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:17.131188 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:17.131162 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:17.258809 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:17.258783 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:17.258946 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:17.258876 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:53:17.269860 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:17.269833 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:17.419708 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:17.419653 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:17.574153 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:17.574125 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:17.911022 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:17.910996 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:18.165501 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:18.165438 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:18.258803 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:18.258786 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:18.258921 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:18.258788 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:18.258921 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:18.258892 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:18.259011 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:18.258947 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:18.325223 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:18.325187 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:19.258186 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:19.258158 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:19.258582 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:19.258271 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:53:20.259005 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:20.258977 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:20.259527 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:20.259089 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:20.259527 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:20.259127 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:20.259527 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:20.259182 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:21.259203 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:21.259173 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:21.259652 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:21.259303 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:53:22.053886 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:22.053850 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:22.259558 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:22.259527 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:22.259912 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:22.259655 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:22.259912 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:22.259703 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:22.259912 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:22.259776 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:23.059827 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:23.059794 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:23.227891 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:23.227859 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:23.258276 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:23.258254 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:23.258473 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:23.258340 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:53:23.396445 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:23.396419 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:24.258430 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:24.258403 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:24.258587 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:24.258563 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:24.261675 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:24.258632 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:24.261675 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:24.259021 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:24.564232 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:24.564183 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pp7mt"] Oct 30 05:53:24.564666 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:24.564312 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:24.564666 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:24.564402 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:53:24.568194 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:24.567807 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4gz57"] Oct 30 05:53:24.568194 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:24.567931 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:24.568194 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:24.568040 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:24.568461 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:24.568436 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kzpqd"] Oct 30 05:53:24.568580 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:24.568550 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:24.568664 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:24.568644 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:25.509257 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:25.509231 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:25.667526 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:25.667501 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:25.816204 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:25.816144 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zqrdw_1b9d8b00-b8a3-4082-ad89-053a60ff7078/dns-node-resolver/0.log" Oct 30 05:53:25.923227 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:25.923131 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:26.216421 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:26.216397 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:26.258418 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:26.258395 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:26.258418 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:26.258414 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:26.258605 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:26.258404 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:26.258605 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:26.258530 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:26.258605 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:26.258571 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:26.258762 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:26.258667 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:53:26.400340 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:26.400315 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:26.544263 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:26.544146 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:27.022128 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:27.022103 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vjrj8_733fbaf7-e75e-46c7-ad59-478b58424087/node-ca/0.log" Oct 30 05:53:27.154681 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:27.154657 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:27.303944 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:27.303918 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:28.032411 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:28.032380 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:28.173713 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:28.173678 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:28.258958 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:28.258927 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:28.259060 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:28.259029 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pp7mt" podUID="a968117a-0683-4844-8936-99ef536894d9" Oct 30 05:53:28.259110 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:28.258927 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:28.259110 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:28.259088 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:28.259187 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:28.259170 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzpqd" podUID="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" Oct 30 05:53:28.259300 ip-10-0-131-226 kubenswrapper[2575]: E1030 05:53:28.259281 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4gz57" podUID="3d693b0d-6752-422d-85d8-e8afb5a111b9" Oct 30 05:53:29.172130 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.172102 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:29.527928 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.527906 2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeReady" Oct 30 05:53:29.528049 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.527986 2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 05:53:29.593742 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.593718 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-j7wwc"] Oct 30 05:53:29.595784 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.595770 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.596977 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.596950 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fxbcf"] Oct 30 05:53:29.598552 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.598537 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fxbcf" Oct 30 05:53:29.614044 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.614022 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 05:53:29.614136 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.614107 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 05:53:29.614190 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.614182 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 05:53:29.615203 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.615174 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x8h66\"" Oct 30 05:53:29.615335 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.615319 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 05:53:29.615430 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.615417 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 05:53:29.618981 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.618967 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 05:53:29.626613 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.626595 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j7wwc"] Oct 30 05:53:29.627037 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.627021 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tvz4j"] Oct 30 05:53:29.628700 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.628687 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:29.646454 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.646431 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fxbcf"] Oct 30 05:53:29.651556 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.651537 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tvz4j"] Oct 30 05:53:29.711903 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.711885 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n69rf\" (UniqueName: \"kubernetes.io/projected/4a82cf91-ad7d-4d8f-b20b-00113772357b-kube-api-access-n69rf\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.712023 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.711926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4a82cf91-ad7d-4d8f-b20b-00113772357b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.712023 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.711946 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xggd\" (UniqueName: \"kubernetes.io/projected/0d76cf54-ecf2-4444-b7a1-024dcb3d215e-kube-api-access-6xggd\") pod \"ingress-canary-fxbcf\" (UID: \"0d76cf54-ecf2-4444-b7a1-024dcb3d215e\") " pod="openshift-ingress-canary/ingress-canary-fxbcf" Oct 30 05:53:29.712023 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.711965 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d76cf54-ecf2-4444-b7a1-024dcb3d215e-cert\") pod \"ingress-canary-fxbcf\" (UID: \"0d76cf54-ecf2-4444-b7a1-024dcb3d215e\") " pod="openshift-ingress-canary/ingress-canary-fxbcf" Oct 30 05:53:29.712126 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.712104 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a82cf91-ad7d-4d8f-b20b-00113772357b-data-volume\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.712161 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.712128 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4a82cf91-ad7d-4d8f-b20b-00113772357b-crio-socket\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.712236 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.712155 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4a82cf91-ad7d-4d8f-b20b-00113772357b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.723786 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.723763 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 05:53:29.723978 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.723963 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 05:53:29.723978 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.723970 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 05:53:29.724124 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.723986 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r6cnz\"" Oct 30 05:53:29.724173 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.724149 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ntdkp\"" Oct 30 05:53:29.812895 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.812842 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-config-volume\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:29.812895 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.812866 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-metrics-tls\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:29.812895 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.812883 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-tmp-dir\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:29.813079 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.812901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4a82cf91-ad7d-4d8f-b20b-00113772357b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.813079 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.812919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xggd\" (UniqueName: \"kubernetes.io/projected/0d76cf54-ecf2-4444-b7a1-024dcb3d215e-kube-api-access-6xggd\") pod \"ingress-canary-fxbcf\" (UID: \"0d76cf54-ecf2-4444-b7a1-024dcb3d215e\") " pod="openshift-ingress-canary/ingress-canary-fxbcf" Oct 30 05:53:29.813079 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.813027 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d76cf54-ecf2-4444-b7a1-024dcb3d215e-cert\") pod \"ingress-canary-fxbcf\" (UID: \"0d76cf54-ecf2-4444-b7a1-024dcb3d215e\") " pod="openshift-ingress-canary/ingress-canary-fxbcf" Oct 30 05:53:29.813257 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.813085 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a82cf91-ad7d-4d8f-b20b-00113772357b-data-volume\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.813257 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.813116 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4a82cf91-ad7d-4d8f-b20b-00113772357b-crio-socket\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.813257 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.813141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4a82cf91-ad7d-4d8f-b20b-00113772357b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.813257 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.813187 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q554\" (UniqueName: \"kubernetes.io/projected/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-kube-api-access-9q554\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:29.813257 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.813254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n69rf\" (UniqueName: \"kubernetes.io/projected/4a82cf91-ad7d-4d8f-b20b-00113772357b-kube-api-access-n69rf\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.813451 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.813354 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4a82cf91-ad7d-4d8f-b20b-00113772357b-crio-socket\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.813451 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.813400 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a82cf91-ad7d-4d8f-b20b-00113772357b-data-volume\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.813534 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.813519 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4a82cf91-ad7d-4d8f-b20b-00113772357b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.816647 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.816629 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4a82cf91-ad7d-4d8f-b20b-00113772357b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.816647 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.816646 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d76cf54-ecf2-4444-b7a1-024dcb3d215e-cert\") pod \"ingress-canary-fxbcf\" (UID: \"0d76cf54-ecf2-4444-b7a1-024dcb3d215e\") " pod="openshift-ingress-canary/ingress-canary-fxbcf" Oct 30 05:53:29.837580 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.837553 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xggd\" (UniqueName: \"kubernetes.io/projected/0d76cf54-ecf2-4444-b7a1-024dcb3d215e-kube-api-access-6xggd\") pod \"ingress-canary-fxbcf\" (UID: \"0d76cf54-ecf2-4444-b7a1-024dcb3d215e\") " pod="openshift-ingress-canary/ingress-canary-fxbcf" Oct 30 05:53:29.837775 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.837758 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n69rf\" (UniqueName: \"kubernetes.io/projected/4a82cf91-ad7d-4d8f-b20b-00113772357b-kube-api-access-n69rf\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.908709 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.908687 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:29.911919 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.911906 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fxbcf" Oct 30 05:53:29.913654 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.913638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-config-volume\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:29.913750 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.913667 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-metrics-tls\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:29.913750 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.913691 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-tmp-dir\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:29.913855 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.913765 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q554\" (UniqueName: \"kubernetes.io/projected/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-kube-api-access-9q554\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:29.914096 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.914076 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-tmp-dir\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:29.914176 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.914163 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-config-volume\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:29.915964 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.915945 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-metrics-tls\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:29.939435 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:29.939407 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q554\" (UniqueName: \"kubernetes.io/projected/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-kube-api-access-9q554\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:30.216834 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.216812 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-j7wwc"] Oct 30 05:53:30.219123 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.219090 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fxbcf"] Oct 30 05:53:30.227809 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:53:30.227786 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d76cf54_ecf2_4444_b7a1_024dcb3d215e.slice/crio-10a9de2e513c930481bea5dcbba061cc7d3b3e7b313fae8938366a308b2381ad WatchSource:0}: Error finding container 10a9de2e513c930481bea5dcbba061cc7d3b3e7b313fae8938366a308b2381ad: Status 404 returned error can't find the container with id 10a9de2e513c930481bea5dcbba061cc7d3b3e7b313fae8938366a308b2381ad Oct 30 05:53:30.238741 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.238722 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:30.258989 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.258966 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:30.258989 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.258986 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:30.259138 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.259041 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:30.292770 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.291950 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:53:30.292770 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.292038 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l4lh5\"" Oct 30 05:53:30.292770 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.292056 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-99gsm\"" Oct 30 05:53:30.292770 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.292132 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xln8v\"" Oct 30 05:53:30.292770 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.292191 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:53:30.292770 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.292296 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:53:30.292770 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.292310 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:53:30.419731 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.419692 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:30.499457 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.499394 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fxbcf" event={"ID":"0d76cf54-ecf2-4444-b7a1-024dcb3d215e","Type":"ContainerStarted","Data":"10a9de2e513c930481bea5dcbba061cc7d3b3e7b313fae8938366a308b2381ad"} Oct 30 05:53:30.500977 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.500955 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7wwc" event={"ID":"4a82cf91-ad7d-4d8f-b20b-00113772357b","Type":"ContainerStarted","Data":"4c6efd9fbf06e2932e94fa8d9873267a1c05b72fa4cbdca2365aa1c49449b8d2"} Oct 30 05:53:30.501086 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.500999 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7wwc" event={"ID":"4a82cf91-ad7d-4d8f-b20b-00113772357b","Type":"ContainerStarted","Data":"9eb84b0ab24ee39fd68b6cb003800fa32779b47f526ec7ade1c050269ab8c78d"} Oct 30 05:53:30.632483 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.631688 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tvz4j"] Oct 30 05:53:30.633705 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:53:30.633674 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9c238a_4ad9_4ad2_98d0_cb4509704ec0.slice/crio-d5a341f09ec907689a352da9c8a7f7710fb23f7b3805b4f48ad615ab58b66cd9 WatchSource:0}: Error finding container d5a341f09ec907689a352da9c8a7f7710fb23f7b3805b4f48ad615ab58b66cd9: Status 404 returned error can't find the container with id d5a341f09ec907689a352da9c8a7f7710fb23f7b3805b4f48ad615ab58b66cd9 Oct 30 05:53:30.661639 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.661612 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:30.906306 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:30.906264 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:31.095154 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.095066 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:31.252821 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.252680 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7"] Oct 30 05:53:31.254675 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.254653 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:31.280079 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.280058 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 05:53:31.292304 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.292150 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 05:53:31.292304 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.292202 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 05:53:31.292304 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.292265 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 05:53:31.292794 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.292684 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 05:53:31.292794 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.292743 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6hg99\"" Oct 30 05:53:31.294320 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.294252 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7"] Oct 30 05:53:31.295364 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.295345 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gfv67"] Oct 30 05:53:31.297422 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.297353 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf"] Oct 30 05:53:31.299103 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.298652 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.299834 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.299815 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.323606 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.323585 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-sys\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.323717 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.323625 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.323717 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.323653 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-textfile\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.323717 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.323679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.323882 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.323749 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19c2a004-564f-442c-a29a-ebcd20f71102-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:31.323941 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.323916 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/19c2a004-564f-442c-a29a-ebcd20f71102-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:31.323993 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.323945 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9cb92418-7659-4dad-9ffc-fa4e29f099a4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.324040 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.323976 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-tls\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.324040 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.324012 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1192599-1c2d-4670-a19f-c4e9faa70c46-metrics-client-ca\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.324147 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.324059 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2g4l\" (UniqueName: \"kubernetes.io/projected/19c2a004-564f-442c-a29a-ebcd20f71102-kube-api-access-t2g4l\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:31.324147 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.324107 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19c2a004-564f-442c-a29a-ebcd20f71102-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:31.324260 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.324242 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgcmh\" (UniqueName: \"kubernetes.io/projected/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-api-access-hgcmh\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.324319 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.324274 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-root\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.324319 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.324300 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-wtmp\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.324382 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.324333 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.324382 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.324363 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqk2s\" (UniqueName: \"kubernetes.io/projected/a1192599-1c2d-4670-a19f-c4e9faa70c46-kube-api-access-sqk2s\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.324479 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.324461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.324532 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.324500 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cb92418-7659-4dad-9ffc-fa4e29f099a4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.324605 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.324585 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-accelerators-collector-config\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.332935 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.332917 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 05:53:31.333014 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.332955 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 05:53:31.333014 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.332990 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rkzps\"" Oct 30 05:53:31.333263 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.333246 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 05:53:31.333328 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.333308 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4fvwv\"" Oct 30 05:53:31.333376 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.333349 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 05:53:31.333376 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.333355 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf"] Oct 30 05:53:31.333772 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.333749 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:31.381759 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.381585 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 05:53:31.381759 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.381724 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.425857 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-sys\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.425882 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.425900 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-textfile\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.425919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.425945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19c2a004-564f-442c-a29a-ebcd20f71102-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.425963 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/19c2a004-564f-442c-a29a-ebcd20f71102-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.425979 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9cb92418-7659-4dad-9ffc-fa4e29f099a4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.425993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-tls\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.426007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1192599-1c2d-4670-a19f-c4e9faa70c46-metrics-client-ca\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.426022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2g4l\" (UniqueName: \"kubernetes.io/projected/19c2a004-564f-442c-a29a-ebcd20f71102-kube-api-access-t2g4l\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.426037 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19c2a004-564f-442c-a29a-ebcd20f71102-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.426073 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgcmh\" (UniqueName: \"kubernetes.io/projected/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-api-access-hgcmh\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.426087 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-root\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.426100 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-wtmp\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.426115 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.426619 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.426133 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqk2s\" (UniqueName: \"kubernetes.io/projected/a1192599-1c2d-4670-a19f-c4e9faa70c46-kube-api-access-sqk2s\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.427858 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.426151 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.427858 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.426177 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cb92418-7659-4dad-9ffc-fa4e29f099a4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.427858 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.426198 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-accelerators-collector-config\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.427858 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.426375 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-sys\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.427858 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.426876 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-wtmp\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.429802 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.429616 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9cb92418-7659-4dad-9ffc-fa4e29f099a4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.442043 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.431968 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1192599-1c2d-4670-a19f-c4e9faa70c46-metrics-client-ca\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.442043 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.433859 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19c2a004-564f-442c-a29a-ebcd20f71102-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:31.442043 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.434039 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-tls\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.442043 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.434115 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19c2a004-564f-442c-a29a-ebcd20f71102-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:31.442043 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.434189 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-root\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.442043 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.434586 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.442043 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.435775 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.442043 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.435907 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cb92418-7659-4dad-9ffc-fa4e29f099a4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.442043 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.437005 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.442043 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.437332 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.442043 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.437619 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-textfile\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.442043 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.440554 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-accelerators-collector-config\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.446387 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.446363 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/19c2a004-564f-442c-a29a-ebcd20f71102-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:31.458767 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.458723 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2g4l\" (UniqueName: \"kubernetes.io/projected/19c2a004-564f-442c-a29a-ebcd20f71102-kube-api-access-t2g4l\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:31.466461 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.466417 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqk2s\" (UniqueName: \"kubernetes.io/projected/a1192599-1c2d-4670-a19f-c4e9faa70c46-kube-api-access-sqk2s\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.468033 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.468010 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgcmh\" (UniqueName: \"kubernetes.io/projected/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-api-access-hgcmh\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.507300 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.507162 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tvz4j" event={"ID":"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0","Type":"ContainerStarted","Data":"d5a341f09ec907689a352da9c8a7f7710fb23f7b3805b4f48ad615ab58b66cd9"} Oct 30 05:53:31.509752 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.509723 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7wwc" event={"ID":"4a82cf91-ad7d-4d8f-b20b-00113772357b","Type":"ContainerStarted","Data":"d759ac4531b47526c27209bafe7e8e6b2e92f40cf50fbc4e9d96c30b72777b91"} Oct 30 05:53:31.560313 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.560283 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:31.569484 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.569281 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:31.619141 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.619107 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:31.626928 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.626747 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:31.717095 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.716993 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:31.890306 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.890284 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7"] Oct 30 05:53:31.944927 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:31.943584 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf"] Oct 30 05:53:31.946240 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:53:31.946118 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb92418_7659_4dad_9ffc_fa4e29f099a4.slice/crio-b928bde3106cba3eea1466a55f0d49cb406d52e6e954301a89b54bb6f7672604 WatchSource:0}: Error finding container b928bde3106cba3eea1466a55f0d49cb406d52e6e954301a89b54bb6f7672604: Status 404 returned error can't find the container with id b928bde3106cba3eea1466a55f0d49cb406d52e6e954301a89b54bb6f7672604 Oct 30 05:53:32.163569 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.163480 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:32.460559 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.460491 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:32.515555 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.515511 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" event={"ID":"9cb92418-7659-4dad-9ffc-fa4e29f099a4","Type":"ContainerStarted","Data":"b928bde3106cba3eea1466a55f0d49cb406d52e6e954301a89b54bb6f7672604"} Oct 30 05:53:32.517039 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.516961 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gfv67" event={"ID":"a1192599-1c2d-4670-a19f-c4e9faa70c46","Type":"ContainerStarted","Data":"4abe5fd4e29c9bf58cc72bfe9707a041ca7d40dd42c667fa4b747e2faeb44a8a"} Oct 30 05:53:32.519226 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.519175 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" event={"ID":"19c2a004-564f-442c-a29a-ebcd20f71102","Type":"ContainerStarted","Data":"dd9e07b0c35feebc4ab7dc8e261bc276647ddc8b5ba366493bad925504ad35e0"} Oct 30 05:53:32.519342 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.519246 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" event={"ID":"19c2a004-564f-442c-a29a-ebcd20f71102","Type":"ContainerStarted","Data":"60e1465790fc54d67582a3630a5f7e8f655e91e2f28fcdd97b3dd908068eac6f"} Oct 30 05:53:32.519342 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.519276 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" event={"ID":"19c2a004-564f-442c-a29a-ebcd20f71102","Type":"ContainerStarted","Data":"99ecccfd8ee64e6996b91730048adcce550c3ab5ed2613bf60d0f73c75a28d91"} Oct 30 05:53:32.665128 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.665084 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:32.766896 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.766826 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:53:32.769764 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.769737 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.793361 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.792713 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 05:53:32.793361 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.792883 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 05:53:32.793361 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.793055 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 05:53:32.793793 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.793706 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 05:53:32.793793 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.793761 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 05:53:32.794043 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.794016 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 05:53:32.794141 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.794119 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2wt58\"" Oct 30 05:53:32.817930 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.817900 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 05:53:32.822567 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.820984 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:53:32.822704 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.822566 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 05:53:32.844905 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.844867 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.845002 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.844910 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-out\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.845002 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.844935 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.845002 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.844966 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.845246 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.845015 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jnx8\" (UniqueName: \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-kube-api-access-2jnx8\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.845246 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.845055 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.845246 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.845075 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.845246 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.845103 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.845246 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.845131 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-web-config\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.845246 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.845170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.845246 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.845235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.845575 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.845267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-volume\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.845575 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.845296 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.856576 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.856488 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:32.887882 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.887723 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 05:53:32.945925 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.945897 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.946041 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.945941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.946041 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.945975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.946041 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.946005 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-web-config\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.946186 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.946046 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.946186 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.946090 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.946186 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.946122 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-volume\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.946186 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.946153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.946186 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.946184 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.946461 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.946230 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-out\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.946461 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.946256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.946461 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.946292 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.946461 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.946334 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnx8\" (UniqueName: \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-kube-api-access-2jnx8\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.947523 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.947493 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.949090 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.948714 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.949090 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.948741 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.951348 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.951060 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.953416 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.953102 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-web-config\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.954163 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.953793 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.954163 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.954126 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.954835 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.954810 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-volume\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.954913 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.954815 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.955350 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.955305 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.955536 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.955477 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-out\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.960264 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.960098 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:32.979656 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:32.979624 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jnx8\" (UniqueName: \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-kube-api-access-2jnx8\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:33.095571 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.095351 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:33.111714 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.111631 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:33.298493 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.298405 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:33.391995 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.391970 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f"] Oct 30 05:53:33.396870 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.396848 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.416908 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.416872 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 05:53:33.417032 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.416931 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 05:53:33.417032 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.417014 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 05:53:33.417202 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.417075 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 05:53:33.417202 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.417102 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 05:53:33.417202 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.417143 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-brovso9nv32hm\"" Oct 30 05:53:33.417202 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.417149 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5x8x2\"" Oct 30 05:53:33.447079 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.446989 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:53:33.448093 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.448071 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f"] Oct 30 05:53:33.449530 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:53:33.449502 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21885d3_9120_4c95_81ad_cdb80fcdf022.slice/crio-2ac00632bbad142c4aa0561179ca2a76a346129f868db53b34119c6704e43339 WatchSource:0}: Error finding container 2ac00632bbad142c4aa0561179ca2a76a346129f868db53b34119c6704e43339: Status 404 returned error can't find the container with id 2ac00632bbad142c4aa0561179ca2a76a346129f868db53b34119c6704e43339 Oct 30 05:53:33.450974 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.450949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e1f5208-4203-46bf-ab5f-8587b9be50de-tmp\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.451077 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.450984 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.451077 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.451002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-grpc-tls\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.451077 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.451031 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.451077 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.451052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44r7s\" (UniqueName: \"kubernetes.io/projected/4e1f5208-4203-46bf-ab5f-8587b9be50de-kube-api-access-44r7s\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.451303 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.451088 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e1f5208-4203-46bf-ab5f-8587b9be50de-metrics-client-ca\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.451303 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.451136 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.451303 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.451154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-tls\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.451303 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.451169 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.529528 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.529493 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-j7wwc" event={"ID":"4a82cf91-ad7d-4d8f-b20b-00113772357b","Type":"ContainerStarted","Data":"375371cf3d95cf3e608c311460da8e144c710ebe0c519e19d8a3c1f38f0ed383"} Oct 30 05:53:33.532631 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.532598 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tvz4j" event={"ID":"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0","Type":"ContainerStarted","Data":"8bd2aacbccee2238167d7f7474616f8511c8b47dab711e6d13d21dffbb54ca9b"} Oct 30 05:53:33.532773 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.532665 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:33.532773 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.532692 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tvz4j" event={"ID":"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0","Type":"ContainerStarted","Data":"4ee3b05a4b2ea79e243f6b4f6a515fd00c02ee83322479b81d3101f861cb20fc"} Oct 30 05:53:33.534769 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.534740 2575 generic.go:356] "Generic (PLEG): container finished" podID="a1192599-1c2d-4670-a19f-c4e9faa70c46" containerID="1032be3ba6dc8255623bd23eaf6d7dff303074262b9fd71a3bcf5518acdc2490" exitCode=0 Oct 30 05:53:33.534873 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.534826 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gfv67" event={"ID":"a1192599-1c2d-4670-a19f-c4e9faa70c46","Type":"ContainerDied","Data":"1032be3ba6dc8255623bd23eaf6d7dff303074262b9fd71a3bcf5518acdc2490"} Oct 30 05:53:33.538685 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.538664 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" event={"ID":"19c2a004-564f-442c-a29a-ebcd20f71102","Type":"ContainerStarted","Data":"6a7dfb4303bffa8040bd99d2f8f888c86d7ad81e3a3619335cef0f95638fc729"} Oct 30 05:53:33.540888 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.540795 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b21885d3-9120-4c95-81ad-cdb80fcdf022","Type":"ContainerStarted","Data":"2ac00632bbad142c4aa0561179ca2a76a346129f868db53b34119c6704e43339"} Oct 30 05:53:33.551497 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.551477 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e1f5208-4203-46bf-ab5f-8587b9be50de-metrics-client-ca\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.551955 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.551936 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.552040 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.551972 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-tls\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.552040 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.552000 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.552154 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.552119 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e1f5208-4203-46bf-ab5f-8587b9be50de-tmp\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.552207 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.552168 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.552207 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.552177 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e1f5208-4203-46bf-ab5f-8587b9be50de-metrics-client-ca\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.552207 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.552198 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-grpc-tls\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.552625 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.552601 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e1f5208-4203-46bf-ab5f-8587b9be50de-tmp\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.552788 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.552766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.552853 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.552825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44r7s\" (UniqueName: \"kubernetes.io/projected/4e1f5208-4203-46bf-ab5f-8587b9be50de-kube-api-access-44r7s\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.556366 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.556345 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-grpc-tls\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.556366 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.556349 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-tls\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.556759 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.556739 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.557286 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.557232 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.558564 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.558531 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.558564 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.558539 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.577681 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.577624 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-j7wwc" podStartSLOduration=2.154027581 podStartE2EDuration="4.577607778s" totalImagesPullingTime="2.423580197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:53:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:33.57698384 +0000 UTC m=+51.840637226" watchObservedRunningTime="2026-10-30 05:53:33.577607778 +0000 UTC m=+51.841261169" Oct 30 05:53:33.585299 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.585275 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44r7s\" (UniqueName: \"kubernetes.io/projected/4e1f5208-4203-46bf-ab5f-8587b9be50de-kube-api-access-44r7s\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:33.677933 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.677873 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-tvz4j" podStartSLOduration=2.543022907 podStartE2EDuration="4.677861222s" totalImagesPullingTime="2.134838315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:33.677702552 +0000 UTC m=+51.941355943" watchObservedRunningTime="2026-10-30 05:53:33.677861222 +0000 UTC m=+51.941514622" Oct 30 05:53:33.722504 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.722462 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" podStartSLOduration=1.4021030620000001 podStartE2EDuration="2.722443524s" totalImagesPullingTime="1.320340462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:33.721812599 +0000 UTC m=+51.985465988" watchObservedRunningTime="2026-10-30 05:53:33.722443524 +0000 UTC m=+51.986096915" Oct 30 05:53:33.734244 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:33.734204 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:34.062005 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:34.061980 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f"] Oct 30 05:53:34.546159 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:34.546129 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" event={"ID":"9cb92418-7659-4dad-9ffc-fa4e29f099a4","Type":"ContainerStarted","Data":"a1e0417ee15f0067ac0616f057763f396741983d4b4bb7c1b9713ed0a6721926"} Oct 30 05:53:34.546681 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:34.546200 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" event={"ID":"9cb92418-7659-4dad-9ffc-fa4e29f099a4","Type":"ContainerStarted","Data":"231d003e0ea77358ded0c14347eec3b48b11de7a2b235d9aeedabcbbf100d524"} Oct 30 05:53:34.546681 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:34.546245 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" event={"ID":"9cb92418-7659-4dad-9ffc-fa4e29f099a4","Type":"ContainerStarted","Data":"2f511c2b0d187c92f9bf9453d87559d91dda486ae63d2773dab53e3a29864caf"} Oct 30 05:53:34.548313 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:34.548287 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gfv67" event={"ID":"a1192599-1c2d-4670-a19f-c4e9faa70c46","Type":"ContainerStarted","Data":"49ff87714fad0a8bfdb0807e07f4623e2d4d8b6821c9ff1daf690169d10e7637"} Oct 30 05:53:34.548409 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:34.548339 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gfv67" event={"ID":"a1192599-1c2d-4670-a19f-c4e9faa70c46","Type":"ContainerStarted","Data":"bbb10d7c1f60e45f9ebbcd10613e1bf3f24ce63e5bdfa1b279720908e750891d"} Oct 30 05:53:34.549724 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:34.549704 2575 generic.go:356] "Generic (PLEG): container finished" podID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerID="1d6fb68ceaffc0f23f79ae294c1490f040e36bda9093c449dd0596e28864d72d" exitCode=0 Oct 30 05:53:34.549804 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:34.549763 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b21885d3-9120-4c95-81ad-cdb80fcdf022","Type":"ContainerDied","Data":"1d6fb68ceaffc0f23f79ae294c1490f040e36bda9093c449dd0596e28864d72d"} Oct 30 05:53:34.550937 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:34.550906 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" event={"ID":"4e1f5208-4203-46bf-ab5f-8587b9be50de","Type":"ContainerStarted","Data":"5c68c91a7c4cdd59270663790f85efd7d979715abade4807e8aeb6fc911b509b"} Oct 30 05:53:34.552367 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:34.552342 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fxbcf" event={"ID":"0d76cf54-ecf2-4444-b7a1-024dcb3d215e","Type":"ContainerStarted","Data":"f3c306767ba7c8c5843ab29a96b952d8611b83a3ab98b81e54f374cc44b9f240"} Oct 30 05:53:34.573624 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:34.573603 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:34.599121 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:34.599075 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" podStartSLOduration=1.8600900949999999 podStartE2EDuration="3.599053157s" totalImagesPullingTime="1.738963062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:34.597661273 +0000 UTC m=+52.861314664" watchObservedRunningTime="2026-10-30 05:53:34.599053157 +0000 UTC m=+52.862706581" Oct 30 05:53:34.707736 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:34.707654 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gfv67" podStartSLOduration=2.24182742 podStartE2EDuration="3.707638943s" totalImagesPullingTime="1.366457492s" totalInitContainerRuntime="99.354031ms" isStatefulPod=true podCreationTimestamp="2026-10-30 05:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:34.706723427 +0000 UTC m=+52.970376815" watchObservedRunningTime="2026-10-30 05:53:34.707638943 +0000 UTC m=+52.971292333" Oct 30 05:53:34.744824 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:34.744796 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:34.944450 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:34.944416 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:35.813787 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.813729 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fxbcf" podStartSLOduration=3.309207151 podStartE2EDuration="6.813708546s" totalImagesPullingTime="3.504501395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:34.756391332 +0000 UTC m=+53.020044722" watchObservedRunningTime="2026-10-30 05:53:35.813708546 +0000 UTC m=+54.077361937" Oct 30 05:53:35.815023 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.814996 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-755798cb46-949x8"] Oct 30 05:53:35.818649 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.818626 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.836586 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.836466 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 05:53:35.836586 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.836551 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 05:53:35.848689 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.848637 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 05:53:35.848689 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.848637 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 05:53:35.848866 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.848785 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fkdst1vuu8j4a\"" Oct 30 05:53:35.849135 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.849108 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lsk6t\"" Oct 30 05:53:35.853626 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.853603 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-755798cb46-949x8"] Oct 30 05:53:35.876718 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.876632 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8b944931-8f1e-4596-a265-ceec8c673872-audit-log\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.876718 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.876689 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8b944931-8f1e-4596-a265-ceec8c673872-metrics-server-audit-profiles\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.876888 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.876800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7mvt\" (UniqueName: \"kubernetes.io/projected/8b944931-8f1e-4596-a265-ceec8c673872-kube-api-access-f7mvt\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.876888 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.876842 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-secret-metrics-server-tls\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.876888 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.876875 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-client-ca-bundle\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.877025 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.876907 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b944931-8f1e-4596-a265-ceec8c673872-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.877025 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.876942 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-secret-metrics-server-client-certs\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.884976 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.884951 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:35.907108 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.905877 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh"] Oct 30 05:53:35.909306 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.909284 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" Oct 30 05:53:35.936572 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.936540 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh"] Oct 30 05:53:35.940849 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.940809 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-72nb9\"" Oct 30 05:53:35.940973 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.940910 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 05:53:35.978378 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.978342 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-secret-metrics-server-client-certs\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.978539 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.978437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8b944931-8f1e-4596-a265-ceec8c673872-audit-log\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.978539 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.978481 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8b944931-8f1e-4596-a265-ceec8c673872-metrics-server-audit-profiles\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.978539 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.978527 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7mvt\" (UniqueName: \"kubernetes.io/projected/8b944931-8f1e-4596-a265-ceec8c673872-kube-api-access-f7mvt\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.978649 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.978555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-secret-metrics-server-tls\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.978649 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.978584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrwht\" (UniqueName: \"kubernetes.io/projected/e2a06249-236e-43d7-b485-0d21b72fb442-kube-api-access-rrwht\") pod \"monitoring-plugin-77f4f76594-rzpvh\" (UID: \"e2a06249-236e-43d7-b485-0d21b72fb442\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" Oct 30 05:53:35.978649 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.978613 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-client-ca-bundle\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.978649 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.978643 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e2a06249-236e-43d7-b485-0d21b72fb442-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rzpvh\" (UID: \"e2a06249-236e-43d7-b485-0d21b72fb442\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" Oct 30 05:53:35.978761 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.978678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b944931-8f1e-4596-a265-ceec8c673872-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.978875 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.978849 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8b944931-8f1e-4596-a265-ceec8c673872-audit-log\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.979522 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.979492 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b944931-8f1e-4596-a265-ceec8c673872-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.980718 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.980318 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8b944931-8f1e-4596-a265-ceec8c673872-metrics-server-audit-profiles\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.981538 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.981513 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-secret-metrics-server-tls\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.981628 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.981574 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-secret-metrics-server-client-certs\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:35.981628 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:35.981606 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-client-ca-bundle\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:36.011766 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.011735 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7mvt\" (UniqueName: \"kubernetes.io/projected/8b944931-8f1e-4596-a265-ceec8c673872-kube-api-access-f7mvt\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:36.079910 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.079885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrwht\" (UniqueName: \"kubernetes.io/projected/e2a06249-236e-43d7-b485-0d21b72fb442-kube-api-access-rrwht\") pod \"monitoring-plugin-77f4f76594-rzpvh\" (UID: \"e2a06249-236e-43d7-b485-0d21b72fb442\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" Oct 30 05:53:36.079986 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.079928 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e2a06249-236e-43d7-b485-0d21b72fb442-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rzpvh\" (UID: \"e2a06249-236e-43d7-b485-0d21b72fb442\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" Oct 30 05:53:36.082407 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.082384 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e2a06249-236e-43d7-b485-0d21b72fb442-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rzpvh\" (UID: \"e2a06249-236e-43d7-b485-0d21b72fb442\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" Oct 30 05:53:36.089245 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.089163 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:36.104203 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.104183 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrwht\" (UniqueName: \"kubernetes.io/projected/e2a06249-236e-43d7-b485-0d21b72fb442-kube-api-access-rrwht\") pod \"monitoring-plugin-77f4f76594-rzpvh\" (UID: \"e2a06249-236e-43d7-b485-0d21b72fb442\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" Oct 30 05:53:36.134740 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.134717 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:36.223015 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.222996 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" Oct 30 05:53:36.307829 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.307767 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:36.454869 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.454829 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-755798cb46-949x8"] Oct 30 05:53:36.469119 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.469099 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:36.539919 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.539896 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh"] Oct 30 05:53:36.540447 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:53:36.540428 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a06249_236e_43d7_b485_0d21b72fb442.slice/crio-f5e209f160727e8b99b61d6c804262aa9b24c8f44a78a301ebba9a3438bcca38 WatchSource:0}: Error finding container f5e209f160727e8b99b61d6c804262aa9b24c8f44a78a301ebba9a3438bcca38: Status 404 returned error can't find the container with id f5e209f160727e8b99b61d6c804262aa9b24c8f44a78a301ebba9a3438bcca38 Oct 30 05:53:36.560706 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.560678 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" event={"ID":"e2a06249-236e-43d7-b485-0d21b72fb442","Type":"ContainerStarted","Data":"f5e209f160727e8b99b61d6c804262aa9b24c8f44a78a301ebba9a3438bcca38"} Oct 30 05:53:36.561838 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.561807 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-755798cb46-949x8" event={"ID":"8b944931-8f1e-4596-a265-ceec8c673872","Type":"ContainerStarted","Data":"df79389aaf838a5971a25bcb2b4eb74ac73ad4a919886a0d8bd965552dcc8105"} Oct 30 05:53:36.564826 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.564806 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b21885d3-9120-4c95-81ad-cdb80fcdf022","Type":"ContainerStarted","Data":"c8e6a5406deb9dd647b72e5013d778ddb66f11eed587beb381249125365a61bd"} Oct 30 05:53:36.564916 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.564854 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b21885d3-9120-4c95-81ad-cdb80fcdf022","Type":"ContainerStarted","Data":"52e85192fd0abb256a2e2c693dcec734355a33580f71e1435f0fe3a128e3ea10"} Oct 30 05:53:36.564916 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.564886 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b21885d3-9120-4c95-81ad-cdb80fcdf022","Type":"ContainerStarted","Data":"107e86592e5a37166c43840d3f66816b76736d29010ed3f147b5b753dffd7c71"} Oct 30 05:53:36.564985 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.564923 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b21885d3-9120-4c95-81ad-cdb80fcdf022","Type":"ContainerStarted","Data":"8e344635d04e78d8822773fe6d87d5e8bb56e101731179354bb2b7c7743143ff"} Oct 30 05:53:36.564985 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.564951 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b21885d3-9120-4c95-81ad-cdb80fcdf022","Type":"ContainerStarted","Data":"dc08fbe805e214511e7b406f3e6f4583f0a548015c82503731efdb53690fa7ab"} Oct 30 05:53:36.566749 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.566731 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" event={"ID":"4e1f5208-4203-46bf-ab5f-8587b9be50de","Type":"ContainerStarted","Data":"436c3a5694805895e1cabf3bd95ddc26312c0d36059f6560b1ce0e9dd50f68df"} Oct 30 05:53:36.566820 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.566777 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" event={"ID":"4e1f5208-4203-46bf-ab5f-8587b9be50de","Type":"ContainerStarted","Data":"c129564daaf938630ba978957ed3d0e36d7797131b001043ee32ec5e6e1e52d3"} Oct 30 05:53:36.566820 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.566803 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" event={"ID":"4e1f5208-4203-46bf-ab5f-8587b9be50de","Type":"ContainerStarted","Data":"810648daa50e1d858bb8c15bc9f883dac21180d433f95e708de067e3c9bf16bf"} Oct 30 05:53:36.696805 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.696774 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:36.716783 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.716747 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7ff58fc747-xt48n"] Oct 30 05:53:36.720145 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.720117 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.742637 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.742616 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 05:53:36.742747 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.742686 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 05:53:36.742747 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.742706 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 05:53:36.742747 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.742720 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 05:53:36.742747 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.742727 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kmt7k\"" Oct 30 05:53:36.759949 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.759908 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 05:53:36.759949 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.759943 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 05:53:36.762200 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.762178 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7ff58fc747-xt48n"] Oct 30 05:53:36.791283 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.791242 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-telemeter-client-tls\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.791418 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.791283 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-serving-certs-ca-bundle\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.791418 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.791311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msst4\" (UniqueName: \"kubernetes.io/projected/e1db1c23-703c-4455-8bbf-23688bec2ede-kube-api-access-msst4\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.791503 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.791435 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-federate-client-tls\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.791546 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.791523 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-secret-telemeter-client\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.791595 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.791552 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.791645 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.791591 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.791645 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.791619 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-metrics-client-ca\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.892964 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.892825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-secret-telemeter-client\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.896507 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.895118 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.896718 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.896699 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.896863 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.896730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-metrics-client-ca\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.896863 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.896756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-telemeter-client-tls\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.896863 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.896782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-serving-certs-ca-bundle\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.896863 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.896806 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msst4\" (UniqueName: \"kubernetes.io/projected/e1db1c23-703c-4455-8bbf-23688bec2ede-kube-api-access-msst4\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.896863 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.896847 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-federate-client-tls\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.897800 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.897781 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-metrics-client-ca\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.897875 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.896185 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.898838 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.898818 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-serving-certs-ca-bundle\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.903008 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.902959 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-telemeter-client-tls\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.904079 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.903545 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.904079 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.904038 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-secret-telemeter-client\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.905754 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.905736 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-federate-client-tls\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:36.919098 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.919062 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:36.926016 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:36.925998 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msst4\" (UniqueName: \"kubernetes.io/projected/e1db1c23-703c-4455-8bbf-23688bec2ede-kube-api-access-msst4\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:37.040769 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:37.040534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:37.392444 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:37.392411 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7ff58fc747-xt48n"] Oct 30 05:53:37.403900 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:53:37.403858 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1db1c23_703c_4455_8bbf_23688bec2ede.slice/crio-8a665e38c56c97a63d1237edf4f5c503a7e771a104c7a7c31453351fee781f78 WatchSource:0}: Error finding container 8a665e38c56c97a63d1237edf4f5c503a7e771a104c7a7c31453351fee781f78: Status 404 returned error can't find the container with id 8a665e38c56c97a63d1237edf4f5c503a7e771a104c7a7c31453351fee781f78 Oct 30 05:53:37.573200 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:37.573123 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" event={"ID":"e1db1c23-703c-4455-8bbf-23688bec2ede","Type":"ContainerStarted","Data":"8a665e38c56c97a63d1237edf4f5c503a7e771a104c7a7c31453351fee781f78"} Oct 30 05:53:37.577410 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:37.577378 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b21885d3-9120-4c95-81ad-cdb80fcdf022","Type":"ContainerStarted","Data":"1095cbe0c9e5b3d06752eab58eab158f24d0059ed18bd84bccdd5b2bb1f01ebb"} Oct 30 05:53:37.580911 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:37.580879 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" event={"ID":"4e1f5208-4203-46bf-ab5f-8587b9be50de","Type":"ContainerStarted","Data":"15016cd5e49ad94592a13d958f3124c7d0e062dcf26b2addfa43e19fd6bd84da"} Oct 30 05:53:37.581022 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:37.580936 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" event={"ID":"4e1f5208-4203-46bf-ab5f-8587b9be50de","Type":"ContainerStarted","Data":"9677d187b36c60502cfcc5d382633443182aa648faa3d28d84ea47886d4c559b"} Oct 30 05:53:37.581022 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:37.580973 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" event={"ID":"4e1f5208-4203-46bf-ab5f-8587b9be50de","Type":"ContainerStarted","Data":"79027300a02c2eff24df06a2954efcbe9791c9c64330b2d99caf3097723fc42d"} Oct 30 05:53:37.581100 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:37.581029 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:37.644420 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:37.644361 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.479936548 podStartE2EDuration="5.644343993s" totalImagesPullingTime="3.151935106s" totalInitContainerRuntime="12.472339ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:37.641666198 +0000 UTC m=+55.905319611" watchObservedRunningTime="2026-10-30 05:53:37.644343993 +0000 UTC m=+55.907997381" Oct 30 05:53:37.709139 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:37.709089 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" podStartSLOduration=1.834564067 podStartE2EDuration="4.709071426s" totalImagesPullingTime="2.874507359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:37.708341887 +0000 UTC m=+55.971995277" watchObservedRunningTime="2026-10-30 05:53:37.709071426 +0000 UTC m=+55.972724817" Oct 30 05:53:38.250087 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.250055 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-66db8fc9d4-pr2ll"] Oct 30 05:53:38.253666 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.253643 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.282286 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.282235 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 05:53:38.282537 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.282506 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 05:53:38.282610 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.282595 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6ht9q\"" Oct 30 05:53:38.286488 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.286471 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 05:53:38.286566 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.286552 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 05:53:38.286566 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.286560 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 05:53:38.286662 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.286590 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 05:53:38.286830 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.286818 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 05:53:38.296372 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.296340 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 05:53:38.300879 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.300860 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66db8fc9d4-pr2ll"] Oct 30 05:53:38.312625 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.312603 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.312736 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.312641 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.312736 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.312683 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.312837 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.312808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp89s\" (UniqueName: \"kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.312907 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.312886 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.312958 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.312923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.312958 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.312947 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d0e685-73c2-4a16-a54f-b6781627a9c8-tmp\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.313060 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.312972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.413883 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.413849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp89s\" (UniqueName: \"kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.414030 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.413896 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.414030 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.413918 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.414030 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.413941 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d0e685-73c2-4a16-a54f-b6781627a9c8-tmp\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.414030 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.413961 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.414238 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.414032 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.414238 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.414058 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.414238 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.414228 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.415137 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.415087 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.415257 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.415176 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.415311 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.415248 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.415676 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.415656 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.415796 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.415779 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d0e685-73c2-4a16-a54f-b6781627a9c8-tmp\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.416690 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.416666 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.417614 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.417591 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.440892 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.440864 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp89s\" (UniqueName: \"kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.472386 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.470363 2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:53:38.474914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.474891 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.497489 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.497468 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 05:53:38.501912 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.501889 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aqh338iohmgj7\"" Oct 30 05:53:38.502041 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.501942 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 05:53:38.502164 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.502148 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 05:53:38.502262 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.502163 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 05:53:38.502262 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.502178 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 05:53:38.502986 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.502969 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 05:53:38.503362 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.503308 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 05:53:38.503362 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.503303 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 05:53:38.503362 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.503357 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 05:53:38.503548 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.503428 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 05:53:38.503548 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.503476 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 05:53:38.503548 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.503486 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9x2zl\"" Oct 30 05:53:38.503807 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.503778 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 05:53:38.503955 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.503938 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 05:53:38.512404 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.512375 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 05:53:38.514909 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.514881 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515023 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.514926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515023 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.514957 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515023 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515005 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515176 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515042 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515176 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515084 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515176 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515115 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515322 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515197 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515322 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515288 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515399 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515320 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515399 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515351 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515399 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515378 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config-out\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515508 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515401 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515508 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515430 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515508 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515466 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hng9h\" (UniqueName: \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-kube-api-access-hng9h\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515508 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515502 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515638 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515525 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515638 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515566 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-web-config\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515638 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515600 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.515638 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.515630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.562813 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.562728 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:53:38.569466 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.569439 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:38.586972 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.586877 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" event={"ID":"e2a06249-236e-43d7-b485-0d21b72fb442","Type":"ContainerStarted","Data":"a3c3c3e63c70d3db1627a194148ff8afd95b28b54c42084bd43669720047077a"} Oct 30 05:53:38.587098 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.587061 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" Oct 30 05:53:38.588880 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.588854 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-755798cb46-949x8" event={"ID":"8b944931-8f1e-4596-a265-ceec8c673872","Type":"ContainerStarted","Data":"681654718bf4e63d2f32912e1317681f277768e65cd58da9cca520ab3b0adc75"} Oct 30 05:53:38.589401 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.589378 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:38.593161 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.593141 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616369 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616448 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616493 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616528 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616545 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config-out\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616560 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616577 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hng9h\" (UniqueName: \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-kube-api-access-hng9h\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616619 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616633 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-web-config\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616678 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.616914 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.617705 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.617705 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616753 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.617705 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.616775 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.618268 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.618012 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.620048 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.620023 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.620185 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.620158 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.620652 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.620631 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.621020 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.620996 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.622242 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.622036 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.622242 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.622181 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.623926 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.622773 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.623926 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.623634 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.623926 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.623834 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.624561 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.624514 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.625427 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.625404 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.625821 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.625801 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config-out\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.626736 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.626614 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.632406 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.626960 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.632491 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.627457 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.632491 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.627611 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.632491 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.631768 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.634728 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.634707 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-web-config\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.657048 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.656988 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-755798cb46-949x8" podStartSLOduration=2.094983061 podStartE2EDuration="3.656970492s" totalImagesPullingTime="1.561987431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:38.655175223 +0000 UTC m=+56.918828612" watchObservedRunningTime="2026-10-30 05:53:38.656970492 +0000 UTC m=+56.920623882" Oct 30 05:53:38.658768 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.658723 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hng9h\" (UniqueName: \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-kube-api-access-hng9h\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.709952 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.709896 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" podStartSLOduration=2.231903487 podStartE2EDuration="3.709883129s" totalImagesPullingTime="1.477979642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:38.7082402 +0000 UTC m=+56.971893590" watchObservedRunningTime="2026-10-30 05:53:38.709883129 +0000 UTC m=+56.973536546" Oct 30 05:53:38.800170 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.800133 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:38.890729 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:38.890703 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66db8fc9d4-pr2ll"] Oct 30 05:53:38.893177 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:53:38.893148 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d0e685_73c2_4a16_a54f_b6781627a9c8.slice/crio-7e97c158e1a2368bd2d8d8113b8eece430e9677d11d4e3c574161f1a8abe121b WatchSource:0}: Error finding container 7e97c158e1a2368bd2d8d8113b8eece430e9677d11d4e3c574161f1a8abe121b: Status 404 returned error can't find the container with id 7e97c158e1a2368bd2d8d8113b8eece430e9677d11d4e3c574161f1a8abe121b Oct 30 05:53:39.072583 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:39.072553 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:39.126293 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:39.126270 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:53:39.126835 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:53:39.126814 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c894e3_ad50_4488_8c5c_5a4bd29ed75a.slice/crio-9c33e658395cfc1d8cabf67c663a80f28379c243e36e7d88f4f3a2f0361567fe WatchSource:0}: Error finding container 9c33e658395cfc1d8cabf67c663a80f28379c243e36e7d88f4f3a2f0361567fe: Status 404 returned error can't find the container with id 9c33e658395cfc1d8cabf67c663a80f28379c243e36e7d88f4f3a2f0361567fe Oct 30 05:53:39.414800 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:39.414757 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:39.596584 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:39.596018 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" event={"ID":"e1db1c23-703c-4455-8bbf-23688bec2ede","Type":"ContainerStarted","Data":"a3701f3302d854e32fcc1c273c316a0539517a1de0e21f6a582c06668c7eb948"} Oct 30 05:53:39.596584 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:39.596085 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" event={"ID":"e1db1c23-703c-4455-8bbf-23688bec2ede","Type":"ContainerStarted","Data":"c2c4b7f817a5325c51cab5d7a3b5c681da3bbea5a877369955737fa6a1f29162"} Oct 30 05:53:39.596584 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:39.596114 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" event={"ID":"e1db1c23-703c-4455-8bbf-23688bec2ede","Type":"ContainerStarted","Data":"84ca2e31e3affda446fc1ebe9426a3b0503ee3acac0110159f57d0653d52be31"} Oct 30 05:53:39.597977 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:39.597947 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66db8fc9d4-pr2ll" event={"ID":"19d0e685-73c2-4a16-a54f-b6781627a9c8","Type":"ContainerStarted","Data":"7e97c158e1a2368bd2d8d8113b8eece430e9677d11d4e3c574161f1a8abe121b"} Oct 30 05:53:39.599733 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:39.599711 2575 generic.go:356] "Generic (PLEG): container finished" podID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerID="41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a" exitCode=0 Oct 30 05:53:39.599846 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:39.599808 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45c894e3-ad50-4488-8c5c-5a4bd29ed75a","Type":"ContainerDied","Data":"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a"} Oct 30 05:53:39.599908 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:39.599880 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45c894e3-ad50-4488-8c5c-5a4bd29ed75a","Type":"ContainerStarted","Data":"9c33e658395cfc1d8cabf67c663a80f28379c243e36e7d88f4f3a2f0361567fe"} Oct 30 05:53:39.663826 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:39.663772 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" podStartSLOduration=2.046296655 podStartE2EDuration="3.663754081s" totalImagesPullingTime="1.617457426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:39.661605586 +0000 UTC m=+57.925258996" watchObservedRunningTime="2026-10-30 05:53:39.663754081 +0000 UTC m=+57.927407471" Oct 30 05:53:40.896142 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:40.896049 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:41.097621 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:41.097573 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:42.555707 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:42.555674 2575 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66db8fc9d4-pr2ll"] Oct 30 05:53:42.613994 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:42.613962 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66db8fc9d4-pr2ll" event={"ID":"19d0e685-73c2-4a16-a54f-b6781627a9c8","Type":"ContainerStarted","Data":"8426d98f2edeb3f6282cca309d46a9ebadc67a3fb93d696eb3ee6abdcd84f845"} Oct 30 05:53:42.668023 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:42.667979 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-66db8fc9d4-pr2ll" podStartSLOduration=1.87912428 podStartE2EDuration="4.667966221s" totalImagesPullingTime="2.788841941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:42.66658398 +0000 UTC m=+60.930237371" watchObservedRunningTime="2026-10-30 05:53:42.667966221 +0000 UTC m=+60.931619609" Oct 30 05:53:43.004753 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:43.004722 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:43.260553 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:43.260521 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:43.554232 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:43.554147 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:43.596513 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:43.596489 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:43.622199 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:43.622167 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45c894e3-ad50-4488-8c5c-5a4bd29ed75a","Type":"ContainerStarted","Data":"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f"} Oct 30 05:53:43.622377 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:43.622262 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45c894e3-ad50-4488-8c5c-5a4bd29ed75a","Type":"ContainerStarted","Data":"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f"} Oct 30 05:53:43.622377 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:43.622302 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45c894e3-ad50-4488-8c5c-5a4bd29ed75a","Type":"ContainerStarted","Data":"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6"} Oct 30 05:53:43.622377 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:43.622337 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45c894e3-ad50-4488-8c5c-5a4bd29ed75a","Type":"ContainerStarted","Data":"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1"} Oct 30 05:53:43.622377 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:43.622371 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45c894e3-ad50-4488-8c5c-5a4bd29ed75a","Type":"ContainerStarted","Data":"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d"} Oct 30 05:53:43.622591 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:43.622406 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45c894e3-ad50-4488-8c5c-5a4bd29ed75a","Type":"ContainerStarted","Data":"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b"} Oct 30 05:53:43.800384 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:43.800358 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:43.837571 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:43.837475 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.785554276 podStartE2EDuration="5.837459178s" totalImagesPullingTime="3.035290605s" totalInitContainerRuntime="16.614297ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:43.835190322 +0000 UTC m=+62.098843711" watchObservedRunningTime="2026-10-30 05:53:43.837459178 +0000 UTC m=+62.101112566" Oct 30 05:53:44.027936 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:44.027903 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:44.287799 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:44.287760 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:44.667688 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:44.667651 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:44.940534 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:44.940431 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:45.344295 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:45.344200 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:45.631177 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:45.631147 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:45.901193 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:45.901128 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:46.077449 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:46.077414 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:47.251791 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:47.251753 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:47.450164 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:47.450138 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:48.024881 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.024848 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:48.025051 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.024913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:48.052738 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.052720 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:53:48.052799 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.052738 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:53:48.058119 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.058089 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:48.058119 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.058104 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:48.125589 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.125567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4cr\" (UniqueName: \"kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr\") pod \"network-check-target-4gz57\" (UID: \"3d693b0d-6752-422d-85d8-e8afb5a111b9\") " pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:48.140253 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.140235 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:53:48.160822 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.160801 2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:53:48.169001 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.168984 2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv4cr\" (UniqueName: \"kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr\") pod \"network-check-target-4gz57\" (UID: \"3d693b0d-6752-422d-85d8-e8afb5a111b9\") " pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:48.290733 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.290677 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-99gsm\"" Oct 30 05:53:48.295735 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.295719 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l4lh5\"" Oct 30 05:53:48.296775 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.296760 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:48.299769 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.299751 2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xln8v\"" Oct 30 05:53:48.301469 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.301456 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:48.306067 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.306052 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:48.569929 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.569860 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:48.614079 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.614056 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kzpqd"] Oct 30 05:53:48.614679 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:53:48.614656 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8fac18d_f521_4f23_ad5e_ec2b5dcac722.slice/crio-3e70d9bad2daea3dc2b749b081fe4ca303502dad59fd6c5a44a83c0e96d2def5 WatchSource:0}: Error finding container 3e70d9bad2daea3dc2b749b081fe4ca303502dad59fd6c5a44a83c0e96d2def5: Status 404 returned error can't find the container with id 3e70d9bad2daea3dc2b749b081fe4ca303502dad59fd6c5a44a83c0e96d2def5 Oct 30 05:53:48.624361 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.624340 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pp7mt"] Oct 30 05:53:48.624793 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:53:48.624772 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda968117a_0683_4844_8936_99ef536894d9.slice/crio-5352dc5415df8189322126ae44a2386e963db3ba01350a24d011f0bc66bf5b0d WatchSource:0}: Error finding container 5352dc5415df8189322126ae44a2386e963db3ba01350a24d011f0bc66bf5b0d: Status 404 returned error can't find the container with id 5352dc5415df8189322126ae44a2386e963db3ba01350a24d011f0bc66bf5b0d Oct 30 05:53:48.628739 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.628716 2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4gz57"] Oct 30 05:53:48.628939 ip-10-0-131-226 kubenswrapper[2575]: W1030 05:53:48.628910 2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d693b0d_6752_422d_85d8_e8afb5a111b9.slice/crio-9b7c3ea94f683de3d7b35991b2b4076f91ef9c792b73f603754b28e83bc1a1e8 WatchSource:0}: Error finding container 9b7c3ea94f683de3d7b35991b2b4076f91ef9c792b73f603754b28e83bc1a1e8: Status 404 returned error can't find the container with id 9b7c3ea94f683de3d7b35991b2b4076f91ef9c792b73f603754b28e83bc1a1e8 Oct 30 05:53:48.642334 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.642311 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pp7mt" event={"ID":"a968117a-0683-4844-8936-99ef536894d9","Type":"ContainerStarted","Data":"5352dc5415df8189322126ae44a2386e963db3ba01350a24d011f0bc66bf5b0d"} Oct 30 05:53:48.643479 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.643457 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4gz57" event={"ID":"3d693b0d-6752-422d-85d8-e8afb5a111b9","Type":"ContainerStarted","Data":"9b7c3ea94f683de3d7b35991b2b4076f91ef9c792b73f603754b28e83bc1a1e8"} Oct 30 05:53:48.644469 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:48.644451 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kzpqd" event={"ID":"f8fac18d-f521-4f23-ad5e-ec2b5dcac722","Type":"ContainerStarted","Data":"3e70d9bad2daea3dc2b749b081fe4ca303502dad59fd6c5a44a83c0e96d2def5"} Oct 30 05:53:49.660377 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:49.660334 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kzpqd" event={"ID":"f8fac18d-f521-4f23-ad5e-ec2b5dcac722","Type":"ContainerStarted","Data":"a15a16d2c0f768ef5f1c65858156d5a10146aee403357ba59b5d0563170e939d"} Oct 30 05:53:50.669556 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:50.669516 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kzpqd" event={"ID":"f8fac18d-f521-4f23-ad5e-ec2b5dcac722","Type":"ContainerStarted","Data":"ec32aa6e0adb3f8a92602991cfc78706abf957c978af9c243e1000fb867e0306"} Oct 30 05:53:50.721813 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:50.721746 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kzpqd" podStartSLOduration=66.796251024 podStartE2EDuration="1m7.721726207s" totalImagesPullingTime="925.475183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:50.720193166 +0000 UTC m=+68.983846559" watchObservedRunningTime="2026-10-30 05:53:50.721726207 +0000 UTC m=+68.985379597" Oct 30 05:53:50.922691 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:50.922608 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:51.135716 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:51.135675 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:52.680051 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:52.680006 2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4gz57" event={"ID":"3d693b0d-6752-422d-85d8-e8afb5a111b9","Type":"ContainerStarted","Data":"51a5cbebb38f835f969419889695ad543d494a970d158a29d773cc503c639762"} Oct 30 05:53:52.680502 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:52.680150 2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:52.724243 ip-10-0-131-226 kubenswrapper[2575]: I1030 05:53:52.723962 2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4gz57" podStartSLOduration=66.389920262 podStartE2EDuration="1m9.723943494s" totalImagesPullingTime="3.334023232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:52.722916262 +0000 UTC m=+70.986569648" watchObservedRunningTime="2026-10-30 05:53:52.723943494 +0000 UTC m=+70.987596886" Oct 30 05:53:53.423892 ip-10-0-131-226 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 05:53:53.433798 ip-10-0-131-226 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 05:53:53.434004 ip-10-0-131-226 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 05:53:53.434187 ip-10-0-131-226 systemd[1]: kubelet.service: Consumed 4.568s CPU time. Oct 30 05:53:53.450761 ip-10-0-131-226 systemd[1]: Starting Kubernetes Kubelet... Oct 30 05:53:53.519547 ip-10-0-131-226 kubenswrapper[6999]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:53:53.519547 ip-10-0-131-226 kubenswrapper[6999]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 05:53:53.519547 ip-10-0-131-226 kubenswrapper[6999]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:53:53.519547 ip-10-0-131-226 kubenswrapper[6999]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:53:53.521504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521434 6999 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:53.521504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521504 6999 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521508 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521511 6999 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521514 6999 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521517 6999 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521520 6999 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521523 6999 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521525 6999 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521528 6999 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521530 6999 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521533 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521536 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521539 6999 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521541 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521544 6999 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521547 6999 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521550 6999 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521552 6999 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521555 6999 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:53.521579 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521557 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521560 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521563 6999 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521566 6999 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521568 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521571 6999 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521573 6999 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521580 6999 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521583 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521586 6999 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521589 6999 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521592 6999 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521594 6999 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521597 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521599 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521602 6999 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521604 6999 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521607 6999 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521610 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521612 6999 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:53.522011 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521615 6999 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521618 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521620 6999 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521623 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521625 6999 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521627 6999 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521630 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521633 6999 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521636 6999 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521638 6999 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521641 6999 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521644 6999 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521646 6999 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521649 6999 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521653 6999 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521655 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521658 6999 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521661 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521663 6999 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521666 6999 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:53.522548 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521669 6999 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521671 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521674 6999 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521677 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521679 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521682 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521685 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521687 6999 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521690 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521692 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521695 6999 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521697 6999 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521700 6999 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521703 6999 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521705 6999 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521707 6999 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521710 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521712 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521715 6999 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:53.523025 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521717 6999 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521720 6999 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521722 6999 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521725 6999 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521728 6999 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521731 6999 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521733 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521736 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521739 6999 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521741 6999 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521744 6999 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521746 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521749 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521751 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521755 6999 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521759 6999 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521763 6999 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521766 6999 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521770 6999 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:53.523504 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521773 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521776 6999 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521782 6999 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521785 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521788 6999 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521790 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521793 6999 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.521796 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522604 6999 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522612 6999 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522615 6999 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522618 6999 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522621 6999 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522623 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522626 6999 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522629 6999 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522632 6999 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522635 6999 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522638 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:53.523957 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522641 6999 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522644 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522647 6999 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522649 6999 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522652 6999 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522655 6999 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522658 6999 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522660 6999 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522663 6999 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522669 6999 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522672 6999 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522674 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522677 6999 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522679 6999 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522682 6999 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522684 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522686 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522689 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522691 6999 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:53.524443 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522696 6999 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522700 6999 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522703 6999 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522705 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522708 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522711 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522714 6999 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522717 6999 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522719 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522722 6999 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522724 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522727 6999 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522731 6999 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522735 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522739 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522744 6999 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522748 6999 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522753 6999 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522757 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522761 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:53.524886 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522764 6999 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522769 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522772 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522784 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522789 6999 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522792 6999 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522797 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522801 6999 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522805 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522809 6999 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522813 6999 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522817 6999 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522821 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522825 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522829 6999 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522833 6999 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522836 6999 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522841 6999 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522848 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522852 6999 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:53.525370 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522856 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522860 6999 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522864 6999 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522868 6999 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522872 6999 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522878 6999 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522882 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522885 6999 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522890 6999 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522893 6999 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522897 6999 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522901 6999 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522905 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522909 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522913 6999 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522917 6999 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522926 6999 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522930 6999 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522935 6999 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:53.525833 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522939 6999 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522943 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522947 6999 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522950 6999 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522955 6999 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522959 6999 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522966 6999 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522974 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522979 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522982 6999 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522985 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522989 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522992 6999 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522994 6999 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.522997 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523000 6999 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523003 6999 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523544 6999 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523554 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:53.526313 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523557 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523560 6999 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523562 6999 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523565 6999 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523568 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523570 6999 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523573 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523576 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523578 6999 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523581 6999 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523584 6999 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523587 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523590 6999 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523593 6999 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523595 6999 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523598 6999 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523601 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523604 6999 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523606 6999 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523609 6999 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:53.526761 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523612 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523614 6999 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523617 6999 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523620 6999 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523624 6999 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523626 6999 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523629 6999 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523631 6999 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523634 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523637 6999 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523639 6999 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523647 6999 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523649 6999 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523652 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523655 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523657 6999 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523660 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523662 6999 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523665 6999 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523668 6999 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:53.527333 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523670 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523673 6999 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523675 6999 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523678 6999 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523681 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523683 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523686 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523688 6999 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523691 6999 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523693 6999 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523696 6999 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523698 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523701 6999 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523703 6999 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523706 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523708 6999 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523712 6999 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523714 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523717 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:53.527932 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523723 6999 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523726 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523730 6999 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523733 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523738 6999 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523741 6999 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523745 6999 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523748 6999 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523751 6999 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523753 6999 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523756 6999 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523759 6999 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523761 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523764 6999 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523766 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523769 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523772 6999 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523775 6999 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523778 6999 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523780 6999 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:53.528684 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523783 6999 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523786 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523788 6999 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523791 6999 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523794 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523796 6999 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523799 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523801 6999 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523804 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523807 6999 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523810 6999 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523812 6999 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523815 6999 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523818 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523821 6999 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523824 6999 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523827 6999 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523830 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523833 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523835 6999 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:53.529257 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523838 6999 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523840 6999 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523843 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523845 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.523848 6999 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.523945 6999 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.523956 6999 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.523963 6999 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.523967 6999 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.523972 6999 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.523976 6999 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.523980 6999 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.523984 6999 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.523987 6999 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.523990 6999 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.523993 6999 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.523996 6999 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.523999 6999 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524002 6999 flags.go:64] FLAG: --cgroup-root="" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524005 6999 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524008 6999 flags.go:64] FLAG: --client-ca-file="" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524012 6999 flags.go:64] FLAG: --cloud-provider="external" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524015 6999 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 05:53:53.530093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524019 6999 flags.go:64] FLAG: --cluster-domain="" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524021 6999 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524025 6999 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524028 6999 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524031 6999 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524036 6999 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524039 6999 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524041 6999 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524045 6999 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524048 6999 flags.go:64] FLAG: --contention-profiling="false" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524051 6999 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524054 6999 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524057 6999 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524060 6999 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524064 6999 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524067 6999 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524069 6999 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524072 6999 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524075 6999 flags.go:64] FLAG: --enable-server="true" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524078 6999 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524083 6999 flags.go:64] FLAG: --event-burst="100" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524086 6999 flags.go:64] FLAG: --event-qps="50" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524089 6999 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524092 6999 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 05:53:53.531060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524095 6999 flags.go:64] FLAG: --eviction-hard="" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524099 6999 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524102 6999 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524105 6999 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524108 6999 flags.go:64] FLAG: --eviction-soft="" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524111 6999 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524115 6999 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524118 6999 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524122 6999 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524125 6999 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524128 6999 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524131 6999 flags.go:64] FLAG: --feature-gates="" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524135 6999 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524138 6999 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524141 6999 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524144 6999 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524147 6999 flags.go:64] FLAG: --healthz-port="10248" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524150 6999 flags.go:64] FLAG: --help="false" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524153 6999 flags.go:64] FLAG: --hostname-override="ip-10-0-131-226.ec2.internal" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524156 6999 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524159 6999 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524162 6999 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524165 6999 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 05:53:53.532131 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524169 6999 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524171 6999 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524174 6999 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524177 6999 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524180 6999 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524183 6999 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524189 6999 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524193 6999 flags.go:64] FLAG: --kube-reserved="" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524196 6999 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524198 6999 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524202 6999 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524204 6999 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524207 6999 flags.go:64] FLAG: --lock-file="" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524225 6999 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524228 6999 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524233 6999 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524239 6999 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524241 6999 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524244 6999 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524247 6999 flags.go:64] FLAG: --logging-format="text" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524250 6999 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524254 6999 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524256 6999 flags.go:64] FLAG: --manifest-url="" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524259 6999 flags.go:64] FLAG: --manifest-url-header="" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524264 6999 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 05:53:53.533136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524267 6999 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524271 6999 flags.go:64] FLAG: --max-pods="110" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524274 6999 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524277 6999 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524280 6999 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524282 6999 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524285 6999 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524288 6999 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524291 6999 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524298 6999 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524301 6999 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524304 6999 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524307 6999 flags.go:64] FLAG: --pod-cidr="" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524310 6999 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524323 6999 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524326 6999 flags.go:64] FLAG: --pods-per-core="0" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524329 6999 flags.go:64] FLAG: --port="10250" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524332 6999 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524335 6999 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c34d0aee64d866a3" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524338 6999 flags.go:64] FLAG: --qos-reserved="" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524341 6999 flags.go:64] FLAG: --read-only-port="10255" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524344 6999 flags.go:64] FLAG: --register-node="true" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524349 6999 flags.go:64] FLAG: --register-with-taints="" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524352 6999 flags.go:64] FLAG: --registry-burst="10" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524355 6999 flags.go:64] FLAG: --registry-qps="5" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524358 6999 flags.go:64] FLAG: --reserved-cpus="" Oct 30 05:53:53.533930 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524361 6999 flags.go:64] FLAG: --reserved-memory="" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524365 6999 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524368 6999 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524371 6999 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524374 6999 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524377 6999 flags.go:64] FLAG: --runonce="false" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524380 6999 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524383 6999 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524386 6999 flags.go:64] FLAG: --seccomp-default="false" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524389 6999 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524391 6999 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524395 6999 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524398 6999 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524401 6999 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524404 6999 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524407 6999 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524410 6999 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524412 6999 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524415 6999 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524418 6999 flags.go:64] FLAG: --system-cgroups="" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524423 6999 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524433 6999 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524437 6999 flags.go:64] FLAG: --tls-cert-file="" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524439 6999 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524443 6999 flags.go:64] FLAG: --tls-min-version="" Oct 30 05:53:53.535055 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524446 6999 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524449 6999 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524452 6999 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524455 6999 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524459 6999 flags.go:64] FLAG: --v="2" Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524463 6999 flags.go:64] FLAG: --version="false" Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524467 6999 flags.go:64] FLAG: --vmodule="" Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524471 6999 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524474 6999 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524651 6999 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524656 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524659 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524662 6999 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524665 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524668 6999 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524670 6999 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524673 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524676 6999 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524678 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524681 6999 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524683 6999 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:53.536184 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524686 6999 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524688 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524691 6999 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524693 6999 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524696 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524698 6999 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524702 6999 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524705 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524708 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524711 6999 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524713 6999 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524716 6999 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524718 6999 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524721 6999 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524725 6999 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524728 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524731 6999 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524734 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524737 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:53.537071 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524740 6999 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524742 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524745 6999 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524747 6999 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524750 6999 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524752 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524754 6999 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524757 6999 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524760 6999 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524762 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524765 6999 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524767 6999 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524770 6999 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524773 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524775 6999 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524778 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524780 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524783 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524785 6999 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524789 6999 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:53.537944 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524792 6999 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524794 6999 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524797 6999 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524799 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524802 6999 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524805 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524807 6999 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524809 6999 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524812 6999 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524814 6999 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524817 6999 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524820 6999 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524822 6999 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524825 6999 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524827 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524830 6999 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524832 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524835 6999 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524838 6999 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:53.538567 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524840 6999 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524844 6999 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524848 6999 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524851 6999 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524854 6999 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524856 6999 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524859 6999 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524862 6999 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524864 6999 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524867 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524869 6999 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524872 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524876 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524878 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524881 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524883 6999 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524886 6999 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524889 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524892 6999 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:53.539240 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524894 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524897 6999 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524900 6999 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524902 6999 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524907 6999 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524909 6999 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524912 6999 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524914 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524917 6999 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524919 6999 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524922 6999 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524925 6999 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524927 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524930 6999 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524932 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524935 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:53.539717 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524938 6999 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.524963 6999 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.524997 6999 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525001 6999 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525005 6999 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525009 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525012 6999 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525015 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525018 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525022 6999 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525024 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525027 6999 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525029 6999 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525032 6999 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525035 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525038 6999 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:53.540097 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525040 6999 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525043 6999 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525046 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525048 6999 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525051 6999 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525054 6999 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525057 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525060 6999 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525062 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525064 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525067 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525070 6999 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525072 6999 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525075 6999 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525077 6999 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525080 6999 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525082 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525085 6999 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525087 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525090 6999 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:53.540488 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525092 6999 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525095 6999 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525097 6999 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525100 6999 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525103 6999 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525106 6999 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525109 6999 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525111 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525115 6999 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525117 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525120 6999 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525123 6999 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525125 6999 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525129 6999 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525133 6999 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525136 6999 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525138 6999 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525142 6999 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525146 6999 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525148 6999 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:53.541050 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525151 6999 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525154 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525156 6999 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525159 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525162 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525164 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525167 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525169 6999 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525172 6999 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525175 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525177 6999 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525179 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525182 6999 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525184 6999 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525187 6999 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525189 6999 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525192 6999 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525196 6999 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525198 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525201 6999 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:53.541663 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525203 6999 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525206 6999 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525222 6999 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525224 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525227 6999 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525230 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525233 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525235 6999 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525238 6999 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525242 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525245 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525248 6999 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525250 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525253 6999 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525256 6999 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525259 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525262 6999 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525265 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525268 6999 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:53.542157 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525270 6999 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525273 6999 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525275 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525278 6999 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525281 6999 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525283 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525286 6999 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525288 6999 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525291 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525294 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525297 6999 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525300 6999 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.525303 6999 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.530752 6999 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.530773 6999 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.530798 6999 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.530806 6999 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530901 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530910 6999 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:53.542665 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530916 6999 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530921 6999 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530925 6999 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530930 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530935 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530939 6999 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530944 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530948 6999 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530952 6999 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530957 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530961 6999 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530965 6999 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530970 6999 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530975 6999 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530979 6999 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530984 6999 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530988 6999 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530993 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.530997 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:53.543181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531002 6999 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531007 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531011 6999 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531015 6999 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531019 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531024 6999 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531028 6999 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531032 6999 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531037 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531041 6999 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531046 6999 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531050 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531054 6999 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531058 6999 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531062 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531067 6999 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531071 6999 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531075 6999 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531089 6999 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531094 6999 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:53.543654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531098 6999 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531102 6999 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531106 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531111 6999 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531115 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531121 6999 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531125 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531130 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531134 6999 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531138 6999 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531142 6999 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531147 6999 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531154 6999 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531160 6999 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531166 6999 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531171 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531176 6999 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531180 6999 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531184 6999 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:53.544160 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531188 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531193 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531198 6999 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531202 6999 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531207 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531228 6999 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531233 6999 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531237 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531241 6999 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531245 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531249 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531254 6999 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531258 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531263 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531267 6999 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531272 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531276 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531280 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531284 6999 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531290 6999 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:53.544654 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531294 6999 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531298 6999 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531302 6999 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531307 6999 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531312 6999 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531316 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531320 6999 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531325 6999 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531329 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531333 6999 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531339 6999 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531346 6999 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531351 6999 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531355 6999 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531359 6999 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531364 6999 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531368 6999 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531373 6999 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531377 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:53.545124 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531381 6999 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531385 6999 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531389 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531394 6999 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531398 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531402 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531406 6999 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.531450 6999 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531675 6999 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531684 6999 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531691 6999 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531696 6999 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531701 6999 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531706 6999 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531711 6999 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531715 6999 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:53.545664 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531719 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531724 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531728 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531732 6999 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531736 6999 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531740 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531744 6999 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531748 6999 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531753 6999 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531757 6999 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531762 6999 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531766 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531770 6999 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531775 6999 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531780 6999 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531784 6999 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531788 6999 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531792 6999 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531796 6999 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:53.546085 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531803 6999 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531808 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531813 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531818 6999 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531822 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531829 6999 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531833 6999 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531837 6999 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531842 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531847 6999 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531851 6999 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531856 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531860 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531864 6999 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531869 6999 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531873 6999 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531877 6999 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531881 6999 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531886 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531890 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:53.546598 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531894 6999 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531898 6999 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531903 6999 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531907 6999 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531911 6999 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531916 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531920 6999 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531925 6999 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531929 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531933 6999 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531937 6999 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531941 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531946 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531950 6999 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531954 6999 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531958 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531963 6999 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531969 6999 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531974 6999 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:53.547135 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531978 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531983 6999 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531987 6999 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531991 6999 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.531996 6999 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532001 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532006 6999 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532010 6999 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532014 6999 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532019 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532023 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532028 6999 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532032 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532036 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532041 6999 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532045 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532050 6999 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532054 6999 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532059 6999 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532063 6999 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:53.547925 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532068 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532072 6999 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532076 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532080 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532084 6999 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532088 6999 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532092 6999 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532097 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532101 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532105 6999 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532111 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532115 6999 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532119 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532123 6999 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532127 6999 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532131 6999 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532135 6999 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532140 6999 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532145 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:53.548617 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532149 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.532188 6999 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532256 6999 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532262 6999 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532267 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532271 6999 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532275 6999 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532279 6999 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532283 6999 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532288 6999 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532292 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532297 6999 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532301 6999 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532305 6999 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532309 6999 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532314 6999 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:53:53.549198 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532318 6999 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532322 6999 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532328 6999 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532335 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532340 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532344 6999 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532348 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532352 6999 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532358 6999 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532362 6999 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532367 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532371 6999 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532376 6999 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532380 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532384 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532388 6999 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532393 6999 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532397 6999 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532401 6999 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:53:53.549826 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532405 6999 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532409 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532414 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532418 6999 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532422 6999 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532426 6999 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532430 6999 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532435 6999 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532439 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532443 6999 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532447 6999 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532451 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532455 6999 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532459 6999 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532464 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532468 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532473 6999 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532477 6999 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532481 6999 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:53:53.550513 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532485 6999 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532490 6999 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532496 6999 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532500 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532505 6999 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532509 6999 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532513 6999 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532519 6999 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532523 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532527 6999 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532531 6999 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532536 6999 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532540 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532545 6999 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532549 6999 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532554 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532558 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532562 6999 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532567 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532571 6999 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:53:53.551181 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532575 6999 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532579 6999 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532584 6999 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532588 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532592 6999 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532597 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532601 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532605 6999 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532609 6999 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532613 6999 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532617 6999 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532621 6999 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532625 6999 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532629 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532636 6999 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532640 6999 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532644 6999 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532648 6999 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532652 6999 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532656 6999 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:53:53.551915 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532661 6999 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532665 6999 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532669 6999 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532673 6999 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532682 6999 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532687 6999 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532692 6999 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532696 6999 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532700 6999 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532704 6999 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532708 6999 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532712 6999 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532716 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:53:53.532721 6999 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.532813 6999 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.534868 6999 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.534938 6999 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.535202 6999 server.go:1014] "Starting client certificate rotation" Oct 30 05:53:53.552516 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.535369 6999 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:53:53.552955 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.535417 6999 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:47:43 +0000 UTC" deadline="2028-04-17 13:11:26.205545523 +0000 UTC" Oct 30 05:53:53.552955 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.535461 6999 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12847h17m32.670086054s" Oct 30 05:53:53.552955 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.535968 6999 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:53:53.552955 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.537578 6999 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:53:53.552955 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.538705 6999 log.go:25] "Validated CRI v1 runtime API" Oct 30 05:53:53.552955 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.540752 6999 log.go:25] "Validated CRI v1 image API" Oct 30 05:53:53.552955 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.541512 6999 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 05:53:53.552955 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.543791 6999 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8487a56f-bfb9-49be-a510-44df142a7f11:/dev/nvme0n1p4 c8c7bd7a-512a-420c-8080-3ad6977584cd:/dev/nvme0n1p3] Oct 30 05:53:53.553180 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.543810 6999 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10a9de2e513c930481bea5dcbba061cc7d3b3e7b313fae8938366a308b2381ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10a9de2e513c930481bea5dcbba061cc7d3b3e7b313fae8938366a308b2381ad/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ac00632bbad142c4aa0561179ca2a76a346129f868db53b34119c6704e43339/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ac00632bbad142c4aa0561179ca2a76a346129f868db53b34119c6704e43339/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b9d2a677fc04337562bb547eb666046282b130b11a94d933c41537e53849fef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b9d2a677fc04337562bb547eb666046282b130b11a94d933c41537e53849fef/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e70d9bad2daea3dc2b749b081fe4ca303502dad59fd6c5a44a83c0e96d2def5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e70d9bad2daea3dc2b749b081fe4ca303502dad59fd6c5a44a83c0e96d2def5/userdata/shm major:0 minor:514 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42fec8443c5340c6d46575e7b9429eda65c9d6f375cb53008cfc97aad156c140/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42fec8443c5340c6d46575e7b9429eda65c9d6f375cb53008cfc97aad156c140/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4abe5fd4e29c9bf58cc72bfe9707a041ca7d40dd42c667fa4b747e2faeb44a8a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4abe5fd4e29c9bf58cc72bfe9707a041ca7d40dd42c667fa4b747e2faeb44a8a/userdata/shm major:0 minor:239 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5352dc5415df8189322126ae44a2386e963db3ba01350a24d011f0bc66bf5b0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5352dc5415df8189322126ae44a2386e963db3ba01350a24d011f0bc66bf5b0d/userdata/shm major:0 minor:518 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c68c91a7c4cdd59270663790f85efd7d979715abade4807e8aeb6fc911b509b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c68c91a7c4cdd59270663790f85efd7d979715abade4807e8aeb6fc911b509b/userdata/shm major:0 minor:319 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/762f4b706be830b590590fef507451cabf71f5ec49bd6e9fde27f3cf55103487/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/762f4b706be830b590590fef507451cabf71f5ec49bd6e9fde27f3cf55103487/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e97c158e1a2368bd2d8d8113b8eece430e9677d11d4e3c574161f1a8abe121b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e97c158e1a2368bd2d8d8113b8eece430e9677d11d4e3c574161f1a8abe121b/userdata/shm major:0 minor:438 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a665e38c56c97a63d1237edf4f5c503a7e771a104c7a7c31453351fee781f78/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a665e38c56c97a63d1237edf4f5c503a7e771a104c7a7c31453351fee781f78/userdata/shm major:0 minor:399 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e0e5c1aa27c35c585015e70714593aa7aee54a1ff842315c94d39062be850f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e0e5c1aa27c35c585015e70714593aa7aee54a1ff842315c94d39062be850f9/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99ecccfd8ee64e6996b91730048adcce550c3ab5ed2613bf60d0f73c75a28d91/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99ecccfd8ee64e6996b91730048adcce550c3ab5ed2613bf60d0f73c75a28d91/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b7c3ea94f683de3d7b35991b2b4076f91ef9c792b73f603754b28e83bc1a1e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b7c3ea94f683de3d7b35991b2b4076f91ef9c792b73f603754b28e83bc1a1e8/userdata/shm major:0 minor:515 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c33e658395cfc1d8cabf67c663a80f28379c243e36e7d88f4f3a2f0361567fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c33e658395cfc1d8cabf67c663a80f28379c243e36e7d88f4f3a2f0361567fe/userdata/shm major:0 minor:452 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9eb84b0ab24ee39fd68b6cb003800fa32779b47f526ec7ade1c050269ab8c78d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9eb84b0ab24ee39fd68b6cb003800fa32779b47f526ec7ade1c050269ab8c78d/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b928bde3106cba3eea1466a55f0d49cb406d52e6e954301a89b54bb6f7672604/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b928bde3106cba3eea1466a55f0d49cb406d52e6e954301a89b54bb6f7672604/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce7779c1faa006e750c121aab96fca0ab9780f2addad1ebbb5767d03f962bb98/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce7779c1faa006e750c121aab96fca0ab9780f2addad1ebbb5767d03f962bb98/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5a341f09ec907689a352da9c8a7f7710fb23f7b3805b4f48ad615ab58b66cd9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5a341f09ec907689a352da9c8a7f7710fb23f7b3805b4f48ad615ab58b66cd9/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da76d95f71989097ae5d4acee3f2d4b25c42927e0b4446c5ca5fb951e07ffef5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da76d95f71989097ae5d4acee3f2d4b25c42927e0b4446c5ca5fb951e07ffef5/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df79389aaf838a5971a25bcb2b4eb74ac73ad4a919886a0d8bd965552dcc8105/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df79389aaf838a5971a25bcb2b4eb74ac73ad4a919886a0d8bd965552dcc8105/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/edcd238349cc00cdebb507db0ad6a83da6bceefa05545386605df62f8634b5ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/edcd238349cc00cdebb507db0ad6a83da6bceefa05545386605df62f8634b5ce/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f01bd5ac744e47ae83a3846db47ffed5b89b9a99c5411662a7fc4a55248dc056/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f01bd5ac744e47ae83a3846db47ffed5b89b9a99c5411662a7fc4a55248dc056/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f5e209f160727e8b99b61d6c804262aa9b24c8f44a78a301ebba9a3438bcca38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f5e209f160727e8b99b61d6c804262aa9b24c8f44a78a301ebba9a3438bcca38/userdata/shm major:0 minor:367 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7e2fe7f2d25db83084f7837e3d33968a456f6b72f0ce64a5c252d74b215044f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7e2fe7f2d25db83084f7837e3d33968a456f6b72f0ce64a5c252d74b215044f/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb64e44a31c75bf1d9c5032529d5a1ac94d9438b1438f51e622a61ec7653921c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb64e44a31c75bf1d9c5032529d5a1ac94d9438b1438f51e622a61ec7653921c/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d76cf54-ecf2-4444-b7a1-024dcb3d215e/volumes/kubernetes.io~projected/kube-api-access-6xggd:{mountpoint:/var/lib/kubelet/pods/0d76cf54-ecf2-4444-b7a1-024dcb3d215e/volumes/kubernetes.io~projected/kube-api-access-6xggd major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d76cf54-ecf2-4444-b7a1-024dcb3d215e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0d76cf54-ecf2-4444-b7a1-024dcb3d215e/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c1aa54-bfc9-4320-915c-59d1701de6d9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/12c1aa54-bfc9-4320-915c-59d1701de6d9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c1aa54-bfc9-4320-915c-59d1701de6d9/volumes/kubernetes.io~projected/kube-api-access-9mvkm:{mountpoint:/var/lib/kubelet/pods/12c1aa54-bfc9-4320-915c-59d1701de6d9/volumes/kubernetes.io~projected/kube-api-access-9mvkm major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12c1aa54-bfc9-4320-915c-59d1701de6d9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/12c1aa54-bfc9-4320-915c-59d1701de6d9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19c2a004-564f-442c-a29a-ebcd20f71102/volumes/kubernetes.io~projected/kube-api-access-t2g4l:{mountpoint:/var/lib/kubelet/pods/19c2a004-564f-442c-a29a-ebcd20f71102/volumes/kubernetes.io~projected/kube-api-access-t2g4l major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19c2a004-564f-442c-a29a-ebcd20f71102/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/19c2a004-564f-442c-a29a-ebcd20f71102/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19c2a004-564f-442c-a29a-ebcd20f71102/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/19c2a004-564f-442c-a29a-ebcd20f71102/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19d0e685-73c2-4a16-a54f-b6781627a9c8/volumes/kubernetes.io~projected/kube-api-access-cp89s:{mountpoint:/var/lib/kubelet/pods/19d0e685-73c2-4a16-a54f-b6781627a9c8/volumes/kubernetes.io~projected/kube-api-access-cp89s major:0 minor:437 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19d0e685-73c2-4a16-a54f-b6781627a9c8/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/19d0e685-73c2-4a16-a54f-b6781627a9c8/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19d0e685-73c2-4a16-a54f-b6781627a9c8/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/19d0e685-73c2-4a16-a54f-b6781627a9c8/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b9d8b00-b8a3-4082-ad89-053a60ff7078/volumes/kubernetes.io~projected/kube-api-access-zdl4j:{mountpoint:/var/lib/kubelet/pods/1b9d8b00-b8a3-4082-ad89-053a60ff7078/volumes/kubernetes.io~projected/kube-api-access-zdl4j major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/26c62aa1-d87b-4997-ae6a-ed1957b6ea73/volumes/kubernetes.io~projected/kube-api-access-gtcl4:{mountpoint:/var/lib/kubelet/pods/26c62aa1-d87b-4997-ae6a-ed1957b6ea73/volumes/kubernetes.io~projected/kube-api-access-gtcl4 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33bd8834-ad74-4370-b805-a6995dd16230/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/33bd8834-ad74-4370-b805-a6995dd16230/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d693b0d-6752-422d-85d8-e8afb5a111b9/volumes/kubernetes.io~projected/kube-api-access-tv4cr:{mountpoint:/var/lib/kubelet/pods/3d693b0d-6752-422d-85d8-e8afb5a111b9/volumes/kubernetes.io~projected/kube-api-access-tv4cr major:0 minor:513 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0/volumes/kubernetes.io~projected/kube-api-access-9q554:{mountpoint:/var/lib/kubelet/pods/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0/volumes/kubernetes.io~projected/kube-api-access-9q554 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0/volumes/kubernetes.io~secret/metrics-tls major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volume-subpaths/web-config/config-reloader/2 major:0 minor:450 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:450 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volume-subpaths/web-config/prometheus/15 major:0 minor:450 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~empty-dir/config-out major:0 minor:447 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~projected/kube-api-access-hng9h:{mountpoint:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~projected/kube-api-access-hng9h major:0 minor:451 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~projected/tls-assets major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/config major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:446 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:445 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:448 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:442 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/web-config major:0 minor:450 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a82cf91-ad7d-4d8f-b20b-00113772357b/volumes/kubernetes.io~projected/kube-api-access-n69rf:{mountpoint:/var/lib/kubelet/pods/4a82cf91-ad7d-4d8f-b20b-00113772357b/volumes/kubernetes.io~projected/kube-api-access-n69rf major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a82cf91-ad7d-4d8f-b20b-00113772357b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/4a82cf91-ad7d-4d8f-b20b-00113772357b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~projected/kube-api-access-44r7s:{mountpoint:/var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~projected/kube-api-access-44r7s major:0 minor:307 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aa4aa89-688f-4a17-b2bc-88dbcd856db8/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/6aa4aa89-688f-4a17-b2bc-88dbcd856db8/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aa4aa89-688f-4a17-b2bc-88dbcd856db8/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/6aa4aa89-688f-4a17-b2bc-88dbcd856db8/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aa4aa89-688f-4a17-b2bc-88dbcd856db8/volumes/kubernetes.io~projected/kube-api-access-lhpvd:{mountpoint:/var/lib/kubelet/pods/6aa4aa89-688f-4a17-b2bc-88dbcd856db8/volumes/kubernetes.io~projected/kube-api-access-lhpvd major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/733fbaf7-e75e-46c7-ad59-478b58424087/volumes/kubernetes.io~projected/kube-api-access-rvn5s:{mountpoint:/var/lib/kubelet/pods/733fbaf7-e75e-46c7-ad59-478b58424087/volumes/kubernetes.io~projected/kube-api-access-rvn5s major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b944931-8f1e-4596-a265-ceec8c673872/volumes/kubernetes.io~projected/kube-api-access-f7mvt:{mountpoint:/var/lib/kubelet/pods/8b944931-8f1e-4596-a265-ceec8c673872/volumes/kubernetes.io~projected/kube-api-access-f7mvt major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b944931-8f1e-4596-a265-ceec8c673872/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/8b944931-8f1e-4596-a265-ceec8c673872/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b944931-8f1e-4596-a265-ceec8c673872/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/8b944931-8f1e-4596-a265-ceec8c673872/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b944931-8f1e-4596-a265-ceec8c673872/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/8b944931-8f1e-4596-a265-ceec8c673872/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cb92418-7659-4dad-9ffc-fa4e29f099a4/volumes/kubernetes.io~projected/kube-api-access-hgcmh:{mountpoint:/var/lib/kubelet/pods/9cb92418-7659-4dad-9ffc-fa4e29f099a4/volumes/kubernetes.io~projected/kube-api-access-hgcmh major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cb92418-7659-4dad-9ffc-fa4e29f099a4/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9cb92418-7659-4dad-9ffc-fa4e29f099a4/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cb92418-7659-4dad-9ffc-fa4e29f099a4/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/9cb92418-7659-4dad-9ffc-fa4e29f099a4/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1192599-1c2d-4670-a19f-c4e9faa70c46/volumes/kubernetes.io~projected/kube-api-access-sqk2s:{mountpoint:/var/lib/kubelet/pods/a1192599-1c2d-4670-a19f-c4e9faa70c46/volumes/kubernetes.io~projected/kube-api-access-sqk2s major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1192599-1c2d-4670-a19f-c4e9faa70c46/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a1192599-1c2d-4670-a19f-c4e9faa70c46/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1192599-1c2d-4670-a19f-c4e9faa70c46/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/a1192599-1c2d-4670-a19f-c4e9faa70c46/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a968117a-0683-4844-8936-99ef536894d9/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/a968117a-0683-4844-8936-99ef536894d9/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:512 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volume-subpaths/web-config/alertmanager/9 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volume-subpaths/web-config/config-reloader/6 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~empty-dir/config-out major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~projected/kube-api-access-2jnx8:{mountpoint:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~projected/kube-api-access-2jnx8 major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~projected/tls-assets major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/config-volume major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/web-config major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb/volumes/kubernetes.io~projected/kube-api-access-b7nmf:{mountpoint:/var/lib/kubelet/pods/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb/volumes/kubernetes.io~projected/kube-api-access-b7nmf major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd135f0d-de15-427c-b9f6-490eaf4a7540/volumes/kubernetes.io~projected/kube-api-access-hdfl5:{mountpoint:/var/lib/kubelet/pods/dd135f0d-de15-427c-b9f6-490eaf4a7540/volumes/kubernetes.io~projected/kube-api-access-hdfl5 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1db1c23-703c-4455-8bbf-23688bec2ede/volumes/kubernetes.io~projected/kube-api-access-msst4:{mountpoint:/var/lib/kubelet/pods/e1db1c23-703c-4455-8bbf-23688bec2ede/volumes/kubernetes.io~projected/kube-api-access-msst4 major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1db1c23-703c-4455-8bbf-23688bec2ede/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/e1db1c23-703c-4455-8bbf-23688bec2ede/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1db1c23-703c-4455-8bbf-23688bec2ede/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/e1db1c23-703c-4455-8bbf-23688bec2ede/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1db1c23-703c-4455-8bbf-23688bec2ede/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e1db1c23-703c-4455-8bbf-23688bec2ede/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1db1c23-703c-4455-8bbf-23688bec2ede/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/e1db1c23-703c-4455-8bbf-23688bec2ede/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2a06249-236e-43d7-b485-0d21b72fb442/volumes/kubernetes.io~projected/kube-api-access-rrwht:{mountpoint:/var/lib/kubelet/pods/e2a06249-236e-43d7-b485-0d21b72fb442/volumes/kubernetes.io~projected/kube-api-access-rrwht major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2a06249-236e-43d7-b485-0d21b72fb442/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/e2a06249-236e-43d7-b485-0d21b72fb442/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e4a59e77-1b37-405f-b787-df3e7e2ecc43/volumes/kubernetes.io~projected/kube-api-access-dcqpr:{mountpoint:/var/lib/kubelet/pods/e4a59e77-1b37-405f-b787-df3e7e2ecc43/volumes/kubernetes.io~projected/kube-api-access-dcqpr major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8fac18d-f521-4f23-ad5e-ec2b5dcac722/volumes/kubernetes.io~projected/kube-api-access-tpt2s:{mountpoint:/var/lib/kubelet/pods/f8fac18d-f521-4f23-ad5e-ec2b5dcac722/volumes/kubernetes.io~projected/kube-api-access-tpt2s major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f8fac18d-f521-4f23-ad5e-ec2b5dcac722/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/f8fac18d-f521-4f23-ad5e-ec2b5dcac722/volumes/kubernetes.io~secret/metrics-certs major:0 minor:508 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/44ed18dac1dce68b8363348aabd0a7e8f17f229b6ad13c40dca1b9b07f39511e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ba0b36c98a2980a43a565fd2efcbe0a1f7a7a6737367397540ccef2f949542d6/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/d3fd5051aa586563002b1d9ab03dc0fba22b7ac467367c3e0d8188f8ed7711f4/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/b540f1e04f1c1b55d8db11005f39aab09a4a639e9f1e839455a957df8aaf2a8e/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d37c240f5f12a09dbe2840985738723f4254c297a84b5a858968facc9c7de4fb/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/295a8957745c8349c8b44bf3ffebb12f84f46630a012be98cf0f4a7bc73b5295/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/469cc4a75044d4b7887ad8bc6c8433b7d58b7e7b9bacf2e4784ec5f1dbf65dbf/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/dd7b2851d493ad4d0e93b44a0a4fd55f34b3abca0a88cbb61695ac81a214d3a0/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/9dbf4b285086e1c02c99197f55556c1b0dcb32d57892b4525e17d7a3d9d17bda/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/d2c4ac390dda5c64aaccbb72f6388eadd2015ddeac53147d9401efe9549a8100/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/ab6140c17a65f506ac37239f2475ce3a4d90beb5ca9b78a20027a8a68962d7f1/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-141:{mountpoint:/var/lib/containers/storage/overlay/3d0ec25e2b4f683d18df05359caca01ab14d70831c1aaae9a06b549cc3c0577b/merged major:0 minor:141 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ec31edff1cdd6cebeb949a972b134af5c80812cf4a8f9f102bab550da267ba5b/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/00c26fe464090cbc4e9019fa4ab99ab5a93f5b0b02dd1211bc1599a19f3e48b2/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/63c65e76c682dc888f926eb71182aac4ec16891fe3024f9f213c9bcc9d9c0a56/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a58ec8a88849aed6d9c817c1510680da388c10d95b39547af98104132ffb2715/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/fd800edbb33525d4fed526c590f1846eccdb10266d58971404cf5986900ee7c9/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/26a8f759baa1790d80c72c1e1c29d8da9b267930f51640c28440b7010d9e62e1/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/0e18dba64b269470c5a56a5197bba49857067d7bb00e99b8fa2bf4887bb231d0/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/01fe3d19c8769940fca0e5723a084b72b01f89067f634cf13be29b01130a7ff7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ea0b5b206b02c7801dee58644750a90a14927108bb35b99f9cf4fabea707eaae/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/b11c44a2eeb5f8e9605067ff63fa1995057b55a9fc3eaa33f000e6066acdf6ef/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/3f2f70dbac7e42f46640cf48eda6531e83af2929afed3612a94d6c4d04111455/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/e68ba17d1cbe6505a79c588472db4a55d8c65a68d2a9400d2ed7e98adbc59926/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/c4cbf69479236dfc275800a7130bbe8082a3293bab31163109ee7965e8a4bcac/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/796e4bdfdec9e9580ed1e1a9e372e7f2337aba1796b6f0edc3bad788cc72e17c/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/92fcf982f1bda38255fb0e5c825a8644b81a4b4e7670e99aaa68659206010e9c/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/c07784963cb0ee9823fe175ab888b285e62e023a89b27a80622e4ee9883edebd/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/e489e7183a302007884d993fb8e4f05883c2781a265ef53300f04e7da297afd0/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/a10888f90100a885e5ba5882b84a5a585c94dcc12d8e9f8f26b1e63d4163e44d/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/c5682947112d7a6263bfbd9c039f3f38e6a4b32822f860ea9d86ed555931ce34/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/a2d0df9892abfe5bc5971900d60839086c0b897553a48641a7bf3741966f9b23/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/cc83e2932d644eca2a4564b2d4457b506a1b7493d48af36c549afdb691eabcea/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/bbda87a113e69214fc4310e3900be0917a55dcb9fe2ada6b01587ff0071aa995/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/a59f03ebe9d152bd4972aa9eac5e5702f28c4acb55b48aae8c7484e459b1716c/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-288:{mountpoint:/var/lib/containers/storage/overlay/74cc1381136fb8661f69f585b4d8c36eeac06a33a07127e9125b2b0151e740d7/merged major:0 minor:288 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/ed05a7478c416c1147d8a6f778d7407bfb89afe5b799b2e366362dec131f0cde/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/67be54cfa66cc0ebd323a58dc6ec291451609bdcf130c7c5eddfbcf3ffd515b5/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/a274e88737d5564927f17daca104e097d6fa2fdf579e48a96e2e0b85b31bfafb/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/893ab217e3933023dfd149f2424cc1569b71322908c7f245327d28a4380e6ac9/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/4e7615317288091b86871fd1b03497579a31dc63c9850eca2511261054618d9f/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/0e25d540a6d984b7101499f7ea2a14fe2c54c9c4c7db1257098e8e7f23149607/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/dd48c2b37215c7ffd85705946bc4c49e339ec9cee39d829ad2729e8bb2297d8b/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/adb351adf43deabb50a6e3aa7c923d62a38470472f606ec44bd48d8b7c6f47d3/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/27ca71f7dd687a48e8c009831e82043e646f5ac0d2cd8f2ee20025c3e65fddab/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/2b579c4f93d31ec68bb057bb07dd95b86e51fa4b4e35a3f03213b5dc7dc0889c/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/ecfe2ca1942814ae73fbe8ea37c7a70975b7586b18b313ce613f8388102e4cf0/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/ef40e2adcc44ab88c0e7126f9252f9ce5637fcc07b9ccb6e84a2896ed33f3a40/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/abb3e8845489a352dfc4678d251618e3664da4de72ec0d63ea99e8f2054764c8/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/dd942d3ea4bdf12d6a86bfae7dafa59998b6ec1718145db8c2599983fa2eceb3/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/cd7e0ef77ebc90b7c47f9e10ec7c3b4b769262b551cb7cf39cc2fbf6ae29d77c/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/b2543863f98b9efb0930e5327ca48d4dffa27b862193be4d67a9ce4a30059492/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/d3e791487d22c91053f75f4174a95d7b99edac041ad8f549f3633b6117243d29/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-401:{mountpoint:/var/lib/containers/storage/overlay/91a2b1a53378fc6f9bc2d4c7bb5640ed58ccd26b9fc75f48c419a7bc41e9478b/merged major:0 minor:401 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/b6c9b15f407fbef14b8d52f4f10dac5cebd02a1249fce38ededc435711dbe21b/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/f453b411073d1113168ac2ce28611e9f0ab0236a8a531a03eeca1b89939a149c/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-413:{mountpoint:/var/lib/containers/storage/overlay/da764ae9ceb9e7cff0b9fe84f3b35ba7a2f83859bdd49fff47626c32d812e9a0/merged major:0 minor:413 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/5cea2f7ffcdd849b296126e247861eba58c222d34343a3ff39ed7a25f57735ae/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-423:{mountpoint:/var/lib/containers/storage/overlay/777293016acd7a10d6483861704998435a33b8dc2563aa5225555f098e6a00a8/merged major:0 minor:423 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/d1ab936d92c958595dc01ded90d475d939724c53af8f42a17eccdc34225fcfd2/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-454:{mountpoint:/var/lib/containers/storage/overlay/7f15c3976bf9b202ba805d8d76f25683b4099326f1fd8f10fc5d23bbab6984e5/merged major:0 minor:454 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/f523dd866e4924d86fd1cb4dee8bbdbcc5bf197eaa030b975fd8ed92da6a0843/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-458:{mountpoint:/var/lib/containers/storage/overlay/615168e6f86134db736e60fa804373335880b1e2f86161e6cdb0f0874051c115/merged major:0 minor:458 fsType:overlay blockSize:0} overlay_0-464:{mountpoint:/var/lib/containers/storage/overlay/63a35283dec0eacf9429fb36ba75d2a49edadb1d762295228ce0544fec5a92b9/merged major:0 minor:464 fsType:overlay blockSize:0} overlay_0-465:{mountpoint:/var/lib/containers/storage/overlay/664ca8ec95198e8088d7d1964c117c0acf90aec6387bd08c59b4a95f932b9663/merged major:0 minor:465 fsType:overlay blockSize:0} overlay_0-470:{mountpoint:/var/lib/containers/storage/overlay/640a0f0bed107723b2a3950caafd492f8f36bbfe9f2d79ad12ebb18bba0a747a/merged major:0 minor:470 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/07e2d73bff1ac5e1f88ae69a7214bc56a10da39fd32b97532a9bf409e597cdf0/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/f7044246a0fa3185b7245da11fa891640071a59a02d5bb5d0282dcbc9aff3dc9/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/850ace4320cf66bda85b7a6dd8d1e939c3ac316d8f046da79aeec0525b8556a1/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-493:{mountpoint:/var/lib/containers/storage/overlay/64063ee3f52a46e52e07e226aa5cb16720a876b091efc4fc2639742851638ef8/merged major:0 minor:493 fsType:overlay blockSize:0} overlay_0-498:{mountpoint:/var/lib/containers/storage/overlay/573e9da3d0e3a9983846afc8479ef3d7e772141d8968a4742a47bc07fbf188ed/merged major:0 minor:498 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/779493670af74130e28d2471e61785aa2f5e8a29c828408161741b0c39ef64e2/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/72c00e9b0f11c6b13f8364ee0c69fcac06135c83d1fd2576a98f0eb59e7561c8/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-520:{mountpoint:/var/lib/containers/storage/overlay/065852b817bbe501071bba4a2f07dce454562ece7884d4ca7378f03a2666518d/merged major:0 minor:520 fsType:overlay blockSize:0} overlay_0-522:{mountpoint:/var/lib/containers/storage/overlay/94095ed9b9fb8e4a5bb9713c72ac2e31e3e46f54863251e33c60670608a90da3/merged major:0 minor:522 fsType:overlay blockSize:0} overlay_0-524:{mountpoint:/var/lib/containers/storage/overlay/3eb31a17243b8a83a1f40f36b4c42f4879f968b816f4ddad50160fd3727b2796/merged major:0 minor:524 fsType:overlay blockSize:0} overlay_0-526:{mountpoint:/var/lib/containers/storage/overlay/7ff97db19b35ccba33c427924d532ddad73ade691ac9ef43c749baa439e761d3/merged major:0 minor:526 fsType:overlay blockSize:0} overlay_0-528:{mountpoint:/var/lib/containers/storage/overlay/e898e2a1b2d16f8d97a8cb9d2fee99cd68ec5cd09d76fe5e3167091e4697140a/merged major:0 minor:528 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/96b2268534612b4982ed9bd87f58f3c2fe2d74b13004c82562735f130a05a434/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-534:{mountpoint:/var/lib/containers/storage/overlay/213503483de79eca78bb61808bc0d1df6a7d47388fe33cf9fb921c24cbec2d3b/merged major:0 minor:534 fsType:overlay blockSize:0} overlay_0-539:{mountpoint:/var/lib/containers/storage/overlay/2a51a4b1ec563ec358f5d9babd03084e80a07ed4b93aa34947d046db27b6663c/merged major:0 minor:539 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/e6fafe89d83cbdc3ca2b146e5ed59516e89950753353fd599e76845204ff277e/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/65e5bf89c8ccbeabc3549e4f15bf2c8d8e6b918510033ea6a70eb3e9cad9783e/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/f6f4dcf60652c98a9e368619444ceea0378b69501cbf9cfb990f6d93f9586484/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/761d3597577ba40974daec5851c7b112ed5c9e488258f7a0848acd5183cdf42d/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/fd07a3d615f024e42378bf530f38a95e2d70c3f8ce0d287af1d1f655ecc775bc/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/9a0b5a73cf3d542cbf2abbd3fb8fc519f15d7696c97e9e8264c0a0905184286a/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/89d79a4f47d160c007429176ed8b2d7011db4ea03739f977872973e38e79e68a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/4597a28faf59b3f2bdde2f8bf8544ff91a7b28ea6c4c96fd96a4383717560a75/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 05:53:53.553504 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.552742 6999 manager.go:217] Machine: {Timestamp:2026-10-30 05:53:53.552294568 +0000 UTC m=+0.061314446 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3116948 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24b653546c15a3f10a5fc601b9fdc8 SystemUUID:ec24b653-546c-15a3-f10a-5fc601b9fdc8 BootID:4b0ac8de-d962-4a3c-88ac-44e919a117d4 Filesystems:[{Device:/run/containers/storage/overlay-containers/f5e209f160727e8b99b61d6c804262aa9b24c8f44a78a301ebba9a3438bcca38/userdata/shm DeviceMajor:0 DeviceMinor:367 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d693b0d-6752-422d-85d8-e8afb5a111b9/volumes/kubernetes.io~projected/kube-api-access-tv4cr DeviceMajor:0 DeviceMinor:513 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/733fbaf7-e75e-46c7-ad59-478b58424087/volumes/kubernetes.io~projected/kube-api-access-rvn5s DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1192599-1c2d-4670-a19f-c4e9faa70c46/volumes/kubernetes.io~projected/kube-api-access-sqk2s DeviceMajor:0 DeviceMinor:235 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:273 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e97c158e1a2368bd2d8d8113b8eece430e9677d11d4e3c574161f1a8abe121b/userdata/shm DeviceMajor:0 DeviceMinor:438 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:441 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dd135f0d-de15-427c-b9f6-490eaf4a7540/volumes/kubernetes.io~projected/kube-api-access-hdfl5 DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4abe5fd4e29c9bf58cc72bfe9707a041ca7d40dd42c667fa4b747e2faeb44a8a/userdata/shm DeviceMajor:0 DeviceMinor:239 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:281 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c68c91a7c4cdd59270663790f85efd7d979715abade4807e8aeb6fc911b509b/userdata/shm DeviceMajor:0 DeviceMinor:319 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e1db1c23-703c-4455-8bbf-23688bec2ede/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:395 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6aa4aa89-688f-4a17-b2bc-88dbcd856db8/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f01bd5ac744e47ae83a3846db47ffed5b89b9a99c5411662a7fc4a55248dc056/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-141 DeviceMajor:0 DeviceMinor:141 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:442 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:444 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:448 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-454 DeviceMajor:0 DeviceMinor:454 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4a82cf91-ad7d-4d8f-b20b-00113772357b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9cb92418-7659-4dad-9ffc-fa4e29f099a4/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:230 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:275 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:279 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b9d2a677fc04337562bb547eb666046282b130b11a94d933c41537e53849fef/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/edcd238349cc00cdebb507db0ad6a83da6bceefa05545386605df62f8634b5ce/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4a82cf91-ad7d-4d8f-b20b-00113772357b/volumes/kubernetes.io~projected/kube-api-access-n69rf DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:447 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:450 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e0e5c1aa27c35c585015e70714593aa7aee54a1ff842315c94d39062be850f9/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb64e44a31c75bf1d9c5032529d5a1ac94d9438b1438f51e622a61ec7653921c/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-401 DeviceMajor:0 DeviceMinor:401 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1192599-1c2d-4670-a19f-c4e9faa70c46/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:231 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:277 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~projected/kube-api-access-44r7s DeviceMajor:0 DeviceMinor:307 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:273 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-493 DeviceMajor:0 DeviceMinor:493 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-526 DeviceMajor:0 DeviceMinor:526 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb/volumes/kubernetes.io~projected/kube-api-access-b7nmf DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6aa4aa89-688f-4a17-b2bc-88dbcd856db8/volumes/kubernetes.io~projected/kube-api-access-lhpvd DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cb92418-7659-4dad-9ffc-fa4e29f099a4/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:229 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/19c2a004-564f-442c-a29a-ebcd20f71102/volumes/kubernetes.io~projected/kube-api-access-t2g4l DeviceMajor:0 DeviceMinor:234 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:445 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-458 DeviceMajor:0 DeviceMinor:458 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-498 DeviceMajor:0 DeviceMinor:498 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d76cf54-ecf2-4444-b7a1-024dcb3d215e/volumes/kubernetes.io~projected/kube-api-access-6xggd DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e1db1c23-703c-4455-8bbf-23688bec2ede/volumes/kubernetes.io~projected/kube-api-access-msst4 DeviceMajor:0 DeviceMinor:398 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42fec8443c5340c6d46575e7b9429eda65c9d6f375cb53008cfc97aad156c140/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d76cf54-ecf2-4444-b7a1-024dcb3d215e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9eb84b0ab24ee39fd68b6cb003800fa32779b47f526ec7ade1c050269ab8c78d/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:274 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/12c1aa54-bfc9-4320-915c-59d1701de6d9/volumes/kubernetes.io~projected/kube-api-access-9mvkm DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:276 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:298 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~projected/kube-api-access-2jnx8 DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:273 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:449 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19c2a004-564f-442c-a29a-ebcd20f71102/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ac00632bbad142c4aa0561179ca2a76a346129f868db53b34119c6704e43339/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:297 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:302 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8b944931-8f1e-4596-a265-ceec8c673872/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:338 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df79389aaf838a5971a25bcb2b4eb74ac73ad4a919886a0d8bd965552dcc8105/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e1db1c23-703c-4455-8bbf-23688bec2ede/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:397 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~projected/kube-api-access-hng9h DeviceMajor:0 DeviceMinor:451 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5352dc5415df8189322126ae44a2386e963db3ba01350a24d011f0bc66bf5b0d/userdata/shm DeviceMajor:0 DeviceMinor:518 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-522 DeviceMajor:0 DeviceMinor:522 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-528 DeviceMajor:0 DeviceMinor:528 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12c1aa54-bfc9-4320-915c-59d1701de6d9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1192599-1c2d-4670-a19f-c4e9faa70c46/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:299 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-423 DeviceMajor:0 DeviceMinor:423 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19d0e685-73c2-4a16-a54f-b6781627a9c8/volumes/kubernetes.io~projected/kube-api-access-cp89s DeviceMajor:0 DeviceMinor:437 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b928bde3106cba3eea1466a55f0d49cb406d52e6e954301a89b54bb6f7672604/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:450 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/33bd8834-ad74-4370-b805-a6995dd16230/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e70d9bad2daea3dc2b749b081fe4ca303502dad59fd6c5a44a83c0e96d2def5/userdata/shm DeviceMajor:0 DeviceMinor:514 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b944931-8f1e-4596-a265-ceec8c673872/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:273 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-413 DeviceMajor:0 DeviceMinor:413 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10a9de2e513c930481bea5dcbba061cc7d3b3e7b313fae8938366a308b2381ad/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:300 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-464 DeviceMajor:0 DeviceMinor:464 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-534 DeviceMajor:0 DeviceMinor:534 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/12c1aa54-bfc9-4320-915c-59d1701de6d9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e4a59e77-1b37-405f-b787-df3e7e2ecc43/volumes/kubernetes.io~projected/kube-api-access-dcqpr DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1b9d8b00-b8a3-4082-ad89-053a60ff7078/volumes/kubernetes.io~projected/kube-api-access-zdl4j DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5a341f09ec907689a352da9c8a7f7710fb23f7b3805b4f48ad615ab58b66cd9/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/19c2a004-564f-442c-a29a-ebcd20f71102/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:233 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da76d95f71989097ae5d4acee3f2d4b25c42927e0b4446c5ca5fb951e07ffef5/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b944931-8f1e-4596-a265-ceec8c673872/volumes/kubernetes.io~projected/kube-api-access-f7mvt DeviceMajor:0 DeviceMinor:343 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c33e658395cfc1d8cabf67c663a80f28379c243e36e7d88f4f3a2f0361567fe/userdata/shm DeviceMajor:0 DeviceMinor:452 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-520 DeviceMajor:0 DeviceMinor:520 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/26c62aa1-d87b-4997-ae6a-ed1957b6ea73/volumes/kubernetes.io~projected/kube-api-access-gtcl4 DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-465 DeviceMajor:0 DeviceMinor:465 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99ecccfd8ee64e6996b91730048adcce550c3ab5ed2613bf60d0f73c75a28d91/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e1db1c23-703c-4455-8bbf-23688bec2ede/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:396 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-470 DeviceMajor:0 DeviceMinor:470 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:278 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a665e38c56c97a63d1237edf4f5c503a7e771a104c7a7c31453351fee781f78/userdata/shm DeviceMajor:0 DeviceMinor:399 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e1f5208-4203-46bf-ab5f-8587b9be50de/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:301 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/19d0e685-73c2-4a16-a54f-b6781627a9c8/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:427 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:450 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b7c3ea94f683de3d7b35991b2b4076f91ef9c792b73f603754b28e83bc1a1e8/userdata/shm DeviceMajor:0 DeviceMinor:515 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0/volumes/kubernetes.io~projected/kube-api-access-9q554 DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2a06249-236e-43d7-b485-0d21b72fb442/volumes/kubernetes.io~projected/kube-api-access-rrwht DeviceMajor:0 DeviceMinor:349 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7e2fe7f2d25db83084f7837e3d33968a456f6b72f0ce64a5c252d74b215044f/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:443 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:446 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a968117a-0683-4844-8936-99ef536894d9/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:512 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6aa4aa89-688f-4a17-b2bc-88dbcd856db8/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f8fac18d-f521-4f23-ad5e-ec2b5dcac722/volumes/kubernetes.io~projected/kube-api-access-tpt2s DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce7779c1faa006e750c121aab96fca0ab9780f2addad1ebbb5767d03f962bb98/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/762f4b706be830b590590fef507451cabf71f5ec49bd6e9fde27f3cf55103487/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-288 DeviceMajor:0 DeviceMinor:288 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b944931-8f1e-4596-a265-ceec8c673872/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:340 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e1db1c23-703c-4455-8bbf-23688bec2ede/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:394 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f8fac18d-f521-4f23-ad5e-ec2b5dcac722/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:508 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:275 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-524 DeviceMajor:0 DeviceMinor:524 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2a06249-236e-43d7-b485-0d21b72fb442/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:344 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19d0e685-73c2-4a16-a54f-b6781627a9c8/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:428 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:440 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:450 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-539 DeviceMajor:0 DeviceMinor:539 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cb92418-7659-4dad-9ffc-fa4e29f099a4/volumes/kubernetes.io~projected/kube-api-access-hgcmh DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:280 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:10a9de2e513c930 MacAddress:0a:b6:ff:ba:9d:44 Speed:10000 Mtu:8901} {Name:2ac00632bbad142 MacAddress:ca:56:f5:81:d4:5e Speed:10000 Mtu:8901} {Name:3e70d9bad2daea3 MacAddress:8a:74:45:c6:d3:ba Speed:10000 Mtu:8901} {Name:5352dc5415df818 MacAddress:7e:92:ec:2f:8b:07 Speed:10000 Mtu:8901} {Name:5c68c91a7c4cdd5 MacAddress:f2:e4:19:8d:9a:d3 Speed:10000 Mtu:8901} {Name:7e97c158e1a2368 MacAddress:d2:f1:a4:46:73:48 Speed:10000 Mtu:8901} {Name:8a665e38c56c97a MacAddress:8e:d5:02:81:e9:75 Speed:10000 Mtu:8901} {Name:99ecccfd8ee64e6 MacAddress:1e:a2:96:7a:78:90 Speed:10000 Mtu:8901} {Name:9b7c3ea94f683de MacAddress:5e:b8:f8:ea:c7:67 Speed:10000 Mtu:8901} {Name:9c33e658395cfc1 MacAddress:1e:ff:cd:f5:0e:16 Speed:10000 Mtu:8901} {Name:9eb84b0ab24ee39 MacAddress:6a:c8:be:29:c9:02 Speed:10000 Mtu:8901} {Name:b928bde3106cba3 MacAddress:8a:ab:d7:c7:b4:01 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fb:3d:46:79 Speed:0 Mtu:9001} {Name:br-int MacAddress:ce:8b:c5:bd:90:ff Speed:0 Mtu:8901} {Name:d5a341f09ec9076 MacAddress:2a:e0:b9:7f:76:e6 Speed:10000 Mtu:8901} {Name:df79389aaf838a5 MacAddress:4e:53:e3:79:e3:6f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fb:3d:46:79 Speed:0 Mtu:9001} {Name:f5e209f160727e8 MacAddress:4e:3c:27:31:82:f3 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:0e:05:c6:a5:08:98 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:c6:b3:54:dc:9a:e9 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 05:53:53.553504 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.553203 6999 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 05:53:53.553504 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.553264 6999 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 05:53:53.553791 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.553580 6999 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 05:53:53.553791 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.553605 6999 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-226.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 05:53:53.553791 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.553720 6999 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 05:53:53.553791 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.553728 6999 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 05:53:53.553791 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.553739 6999 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:53:53.553791 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.553750 6999 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:53:53.553955 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.553819 6999 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 05:53:53.553983 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.553973 6999 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 05:53:53.554081 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.554071 6999 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 05:53:53.554109 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.554105 6999 kubelet.go:493] "Attempting to sync node with API server" Oct 30 05:53:53.554136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.554114 6999 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 05:53:53.554136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.554124 6999 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 05:53:53.554136 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.554132 6999 kubelet.go:405] "Adding apiserver pod source" Oct 30 05:53:53.554265 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.554139 6999 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 05:53:53.555070 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.555055 6999 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 05:53:53.555070 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.555067 6999 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:53:53.555598 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.555577 6999 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 05:53:53.558748 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.558735 6999 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 05:53:53.558803 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.558749 6999 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:53:53.558856 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.558847 6999 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 05:53:53.559035 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559020 6999 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 05:53:53.559065 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559054 6999 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 05:53:53.559167 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559158 6999 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 05:53:53.559205 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559174 6999 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 05:53:53.559205 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559180 6999 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 05:53:53.559205 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559185 6999 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 05:53:53.559205 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559191 6999 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 05:53:53.559324 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559206 6999 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 05:53:53.559324 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559312 6999 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 05:53:53.559324 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559317 6999 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 05:53:53.559324 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559324 6999 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 05:53:53.559434 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559330 6999 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 05:53:53.559434 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559342 6999 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 05:53:53.559434 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559351 6999 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 05:53:53.559434 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559376 6999 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 05:53:53.559434 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.559382 6999 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 05:53:53.562676 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.562661 6999 server.go:1263] "Started kubelet" Oct 30 05:53:53.562777 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.562702 6999 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 05:53:53.563193 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.562817 6999 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 05:53:53.563193 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.562906 6999 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 05:53:53.563325 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.563204 6999 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 05:53:53.563258 ip-10-0-131-226 systemd[1]: Started Kubernetes Kubelet. Oct 30 05:53:53.566623 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.566594 6999 server.go:317] "Adding debug handlers to kubelet server" Oct 30 05:53:53.569988 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.569470 6999 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 05:53:53.569988 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.569501 6999 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 05:53:53.569988 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.569558 6999 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:47:45 +0000 UTC" deadline="2028-04-12 23:02:07.506836245 +0000 UTC" Oct 30 05:53:53.569988 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.569575 6999 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12737h8m13.937264431s" Oct 30 05:53:53.569988 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:53.569604 6999 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-226.ec2.internal\" not found" Oct 30 05:53:53.569988 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.569627 6999 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 05:53:53.569988 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.569640 6999 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 05:53:53.569988 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.569699 6999 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 05:53:53.571322 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.571302 6999 factory.go:153] Registering CRI-O factory Oct 30 05:53:53.571322 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.571324 6999 factory.go:223] Registration of the crio container factory successfully Oct 30 05:53:53.571463 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.571436 6999 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 05:53:53.571463 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.571447 6999 factory.go:55] Registering systemd factory Oct 30 05:53:53.571463 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.571455 6999 factory.go:223] Registration of the systemd container factory successfully Oct 30 05:53:53.571624 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.571473 6999 factory.go:103] Registering Raw factory Oct 30 05:53:53.571624 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.571486 6999 manager.go:1199] Started watching for new ooms in manager Oct 30 05:53:53.571830 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.571816 6999 manager.go:319] Starting recovery of all containers Oct 30 05:53:53.573504 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573469 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b944931-8f1e-4596-a265-ceec8c673872" volumeName="kubernetes.io/projected/8b944931-8f1e-4596-a265-ceec8c673872-kube-api-access-f7mvt" seLinuxMountContext="" Oct 30 05:53:53.573596 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573527 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a968117a-0683-4844-8936-99ef536894d9" volumeName="kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret" seLinuxMountContext="" Oct 30 05:53:53.573596 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573544 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21885d3-9120-4c95-81ad-cdb80fcdf022" volumeName="kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-volume" seLinuxMountContext="" Oct 30 05:53:53.573596 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573555 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-web-config" seLinuxMountContext="" Oct 30 05:53:53.573596 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573569 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d76cf54-ecf2-4444-b7a1-024dcb3d215e" volumeName="kubernetes.io/secret/0d76cf54-ecf2-4444-b7a1-024dcb3d215e-cert" seLinuxMountContext="" Oct 30 05:53:53.573596 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573582 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26c62aa1-d87b-4997-ae6a-ed1957b6ea73" volumeName="kubernetes.io/projected/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-kube-api-access-gtcl4" seLinuxMountContext="" Oct 30 05:53:53.573596 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573593 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573604 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573619 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c1aa54-bfc9-4320-915c-59d1701de6d9" volumeName="kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-env-overrides" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573630 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c1aa54-bfc9-4320-915c-59d1701de6d9" volumeName="kubernetes.io/projected/12c1aa54-bfc9-4320-915c-59d1701de6d9-kube-api-access-9mvkm" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573642 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33bd8834-ad74-4370-b805-a6995dd16230" volumeName="kubernetes.io/configmap/33bd8834-ad74-4370-b805-a6995dd16230-konnectivity-ca" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573653 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e1f5208-4203-46bf-ab5f-8587b9be50de" volumeName="kubernetes.io/empty-dir/4e1f5208-4203-46bf-ab5f-8587b9be50de-tmp" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573664 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb92418-7659-4dad-9ffc-fa4e29f099a4" volumeName="kubernetes.io/secret/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573674 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21885d3-9120-4c95-81ad-cdb80fcdf022" volumeName="kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-web-config" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573691 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19c2a004-564f-442c-a29a-ebcd20f71102" volumeName="kubernetes.io/projected/19c2a004-564f-442c-a29a-ebcd20f71102-kube-api-access-t2g4l" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573702 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19d0e685-73c2-4a16-a54f-b6781627a9c8" volumeName="kubernetes.io/empty-dir/19d0e685-73c2-4a16-a54f-b6781627a9c8-tmp" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573713 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26c62aa1-d87b-4997-ae6a-ed1957b6ea73" volumeName="kubernetes.io/configmap/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-daemon-config" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573725 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d693b0d-6752-422d-85d8-e8afb5a111b9" volumeName="kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573735 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573747 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573775 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config-out" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573783 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573790 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19d0e685-73c2-4a16-a54f-b6781627a9c8" volumeName="kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573797 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b9d8b00-b8a3-4082-ad89-053a60ff7078" volumeName="kubernetes.io/projected/1b9d8b00-b8a3-4082-ad89-053a60ff7078-kube-api-access-zdl4j" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573804 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573812 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573819 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="733fbaf7-e75e-46c7-ad59-478b58424087" volumeName="kubernetes.io/projected/733fbaf7-e75e-46c7-ad59-478b58424087-kube-api-access-rvn5s" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573826 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1192599-1c2d-4670-a19f-c4e9faa70c46" volumeName="kubernetes.io/empty-dir/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-textfile" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573838 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8ae8ad0-aec2-4513-a5b5-75e6246d92fb" volumeName="kubernetes.io/configmap/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-iptables-alerter-script" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573845 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="26c62aa1-d87b-4997-ae6a-ed1957b6ea73" volumeName="kubernetes.io/configmap/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-cni-binary-copy" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573853 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33bd8834-ad74-4370-b805-a6995dd16230" volumeName="kubernetes.io/secret/33bd8834-ad74-4370-b805-a6995dd16230-agent-certs" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573860 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573867 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21885d3-9120-4c95-81ad-cdb80fcdf022" volumeName="kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-metrics-client-ca" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573874 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1db1c23-703c-4455-8bbf-23688bec2ede" volumeName="kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573882 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19d0e685-73c2-4a16-a54f-b6781627a9c8" volumeName="kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573888 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1db1c23-703c-4455-8bbf-23688bec2ede" volumeName="kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-federate-client-tls" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573898 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b9d8b00-b8a3-4082-ad89-053a60ff7078" volumeName="kubernetes.io/empty-dir/1b9d8b00-b8a3-4082-ad89-053a60ff7078-tmp-dir" seLinuxMountContext="" Oct 30 05:53:53.573891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573905 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a82cf91-ad7d-4d8f-b20b-00113772357b" volumeName="kubernetes.io/empty-dir/4a82cf91-ad7d-4d8f-b20b-00113772357b-data-volume" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573913 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21885d3-9120-4c95-81ad-cdb80fcdf022" volumeName="kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573929 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8ae8ad0-aec2-4513-a5b5-75e6246d92fb" volumeName="kubernetes.io/projected/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-kube-api-access-b7nmf" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573937 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9c238a-4ad9-4ad2-98d0-cb4509704ec0" volumeName="kubernetes.io/projected/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-kube-api-access-9q554" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573945 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573952 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb92418-7659-4dad-9ffc-fa4e29f099a4" volumeName="kubernetes.io/configmap/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573958 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1db1c23-703c-4455-8bbf-23688bec2ede" volumeName="kubernetes.io/projected/e1db1c23-703c-4455-8bbf-23688bec2ede-kube-api-access-msst4" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573966 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19d0e685-73c2-4a16-a54f-b6781627a9c8" volumeName="kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573972 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19d0e685-73c2-4a16-a54f-b6781627a9c8" volumeName="kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573980 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a82cf91-ad7d-4d8f-b20b-00113772357b" volumeName="kubernetes.io/secret/4a82cf91-ad7d-4d8f-b20b-00113772357b-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573987 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb92418-7659-4dad-9ffc-fa4e29f099a4" volumeName="kubernetes.io/empty-dir/9cb92418-7659-4dad-9ffc-fa4e29f099a4-volume-directive-shadow" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.573993 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4a59e77-1b37-405f-b787-df3e7e2ecc43" volumeName="kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574000 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4a59e77-1b37-405f-b787-df3e7e2ecc43" volumeName="kubernetes.io/projected/e4a59e77-1b37-405f-b787-df3e7e2ecc43-kube-api-access-dcqpr" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574007 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21885d3-9120-4c95-81ad-cdb80fcdf022" volumeName="kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574014 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c1aa54-bfc9-4320-915c-59d1701de6d9" volumeName="kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovnkube-script-lib" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574021 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574027 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e1f5208-4203-46bf-ab5f-8587b9be50de" volumeName="kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574043 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aa4aa89-688f-4a17-b2bc-88dbcd856db8" volumeName="kubernetes.io/empty-dir/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-tmp" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574051 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1192599-1c2d-4670-a19f-c4e9faa70c46" volumeName="kubernetes.io/configmap/a1192599-1c2d-4670-a19f-c4e9faa70c46-metrics-client-ca" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574068 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9c238a-4ad9-4ad2-98d0-cb4509704ec0" volumeName="kubernetes.io/configmap/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-config-volume" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574075 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb92418-7659-4dad-9ffc-fa4e29f099a4" volumeName="kubernetes.io/secret/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574082 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1db1c23-703c-4455-8bbf-23688bec2ede" volumeName="kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-secret-telemeter-client" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574090 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1db1c23-703c-4455-8bbf-23688bec2ede" volumeName="kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-telemeter-client-tls" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574098 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c1aa54-bfc9-4320-915c-59d1701de6d9" volumeName="kubernetes.io/secret/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574105 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-tls-assets" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574115 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" volumeName="kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574123 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21885d3-9120-4c95-81ad-cdb80fcdf022" volumeName="kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574130 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e1f5208-4203-46bf-ab5f-8587b9be50de" volumeName="kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574138 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aa4aa89-688f-4a17-b2bc-88dbcd856db8" volumeName="kubernetes.io/projected/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-kube-api-access-lhpvd" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574145 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1192599-1c2d-4670-a19f-c4e9faa70c46" volumeName="kubernetes.io/projected/a1192599-1c2d-4670-a19f-c4e9faa70c46-kube-api-access-sqk2s" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574151 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1192599-1c2d-4670-a19f-c4e9faa70c46" volumeName="kubernetes.io/configmap/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574159 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a82cf91-ad7d-4d8f-b20b-00113772357b" volumeName="kubernetes.io/configmap/4a82cf91-ad7d-4d8f-b20b-00113772357b-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574191 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19c2a004-564f-442c-a29a-ebcd20f71102" volumeName="kubernetes.io/configmap/19c2a004-564f-442c-a29a-ebcd20f71102-metrics-client-ca" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574199 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19c2a004-564f-442c-a29a-ebcd20f71102" volumeName="kubernetes.io/secret/19c2a004-564f-442c-a29a-ebcd20f71102-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574224 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2a06249-236e-43d7-b485-0d21b72fb442" volumeName="kubernetes.io/secret/e2a06249-236e-43d7-b485-0d21b72fb442-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574235 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4a59e77-1b37-405f-b787-df3e7e2ecc43" volumeName="kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cni-binary-copy" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574243 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19d0e685-73c2-4a16-a54f-b6781627a9c8" volumeName="kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574250 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e1f5208-4203-46bf-ab5f-8587b9be50de" volumeName="kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-grpc-tls" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574258 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e1f5208-4203-46bf-ab5f-8587b9be50de" volumeName="kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574265 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21885d3-9120-4c95-81ad-cdb80fcdf022" volumeName="kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-main-db" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574272 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21885d3-9120-4c95-81ad-cdb80fcdf022" volumeName="kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-cluster-tls-config" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574280 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd135f0d-de15-427c-b9f6-490eaf4a7540" volumeName="kubernetes.io/projected/dd135f0d-de15-427c-b9f6-490eaf4a7540-kube-api-access-hdfl5" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574287 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e4a59e77-1b37-405f-b787-df3e7e2ecc43" volumeName="kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574295 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574302 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b944931-8f1e-4596-a265-ceec8c673872" volumeName="kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574309 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb92418-7659-4dad-9ffc-fa4e29f099a4" volumeName="kubernetes.io/projected/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-api-access-hgcmh" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574316 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1192599-1c2d-4670-a19f-c4e9faa70c46" volumeName="kubernetes.io/secret/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-tls" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574324 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e1f5208-4203-46bf-ab5f-8587b9be50de" volumeName="kubernetes.io/projected/4e1f5208-4203-46bf-ab5f-8587b9be50de-kube-api-access-44r7s" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574332 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="733fbaf7-e75e-46c7-ad59-478b58424087" volumeName="kubernetes.io/configmap/733fbaf7-e75e-46c7-ad59-478b58424087-serviceca" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574340 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b944931-8f1e-4596-a265-ceec8c673872" volumeName="kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-client-ca-bundle" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574348 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21885d3-9120-4c95-81ad-cdb80fcdf022" volumeName="kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-out" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574357 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21885d3-9120-4c95-81ad-cdb80fcdf022" volumeName="kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-kube-api-access-2jnx8" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574366 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f8fac18d-f521-4f23-ad5e-ec2b5dcac722" volumeName="kubernetes.io/projected/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-kube-api-access-tpt2s" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574373 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19d0e685-73c2-4a16-a54f-b6781627a9c8" volumeName="kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574382 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574389 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574397 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e1f5208-4203-46bf-ab5f-8587b9be50de" volumeName="kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574404 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21885d3-9120-4c95-81ad-cdb80fcdf022" volumeName="kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574410 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1db1c23-703c-4455-8bbf-23688bec2ede" volumeName="kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-metrics-client-ca" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574418 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d76cf54-ecf2-4444-b7a1-024dcb3d215e" volumeName="kubernetes.io/projected/0d76cf54-ecf2-4444-b7a1-024dcb3d215e-kube-api-access-6xggd" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574426 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9c238a-4ad9-4ad2-98d0-cb4509704ec0" volumeName="kubernetes.io/empty-dir/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-tmp-dir" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574433 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b944931-8f1e-4596-a265-ceec8c673872" volumeName="kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574440 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21885d3-9120-4c95-81ad-cdb80fcdf022" volumeName="kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-tls-assets" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574447 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-grpc-tls" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574456 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb92418-7659-4dad-9ffc-fa4e29f099a4" volumeName="kubernetes.io/configmap/9cb92418-7659-4dad-9ffc-fa4e29f099a4-metrics-client-ca" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574466 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b21885d3-9120-4c95-81ad-cdb80fcdf022" volumeName="kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574473 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1192599-1c2d-4670-a19f-c4e9faa70c46" volumeName="kubernetes.io/secret/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574481 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19d0e685-73c2-4a16-a54f-b6781627a9c8" volumeName="kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574488 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-kube-api-access-hng9h" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574494 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e1f5208-4203-46bf-ab5f-8587b9be50de" volumeName="kubernetes.io/configmap/4e1f5208-4203-46bf-ab5f-8587b9be50de-metrics-client-ca" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574501 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e1f5208-4203-46bf-ab5f-8587b9be50de" volumeName="kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574510 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b944931-8f1e-4596-a265-ceec8c673872" volumeName="kubernetes.io/empty-dir/8b944931-8f1e-4596-a265-ceec8c673872-audit-log" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574517 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1db1c23-703c-4455-8bbf-23688bec2ede" volumeName="kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574525 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12c1aa54-bfc9-4320-915c-59d1701de6d9" volumeName="kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovnkube-config" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574532 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b944931-8f1e-4596-a265-ceec8c673872" volumeName="kubernetes.io/configmap/8b944931-8f1e-4596-a265-ceec8c673872-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574550 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a82cf91-ad7d-4d8f-b20b-00113772357b" volumeName="kubernetes.io/projected/4a82cf91-ad7d-4d8f-b20b-00113772357b-kube-api-access-n69rf" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574557 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1db1c23-703c-4455-8bbf-23688bec2ede" volumeName="kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574565 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19c2a004-564f-442c-a29a-ebcd20f71102" volumeName="kubernetes.io/secret/19c2a004-564f-442c-a29a-ebcd20f71102-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574571 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b944931-8f1e-4596-a265-ceec8c673872" volumeName="kubernetes.io/configmap/8b944931-8f1e-4596-a265-ceec8c673872-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574581 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2a06249-236e-43d7-b485-0d21b72fb442" volumeName="kubernetes.io/projected/e2a06249-236e-43d7-b485-0d21b72fb442-kube-api-access-rrwht" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574589 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e9c238a-4ad9-4ad2-98d0-cb4509704ec0" volumeName="kubernetes.io/secret/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-metrics-tls" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574596 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-db" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574603 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" volumeName="kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574610 6999 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aa4aa89-688f-4a17-b2bc-88dbcd856db8" volumeName="kubernetes.io/empty-dir/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-tuned" seLinuxMountContext="" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574617 6999 reconstruct.go:97] "Volume reconstruction finished" Oct 30 05:53:53.575557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.574623 6999 reconciler.go:29] "Reconciler: start to sync state" Oct 30 05:53:53.598919 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.598854 6999 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 05:53:53.600400 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.600365 6999 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 05:53:53.600590 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.600571 6999 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 05:53:53.600682 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.600615 6999 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 05:53:53.600741 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:53.600674 6999 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 05:53:53.612773 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.612741 6999 generic.go:356] "Generic (PLEG): container finished" podID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerID="41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a" exitCode=0 Oct 30 05:53:53.621850 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.621824 6999 generic.go:356] "Generic (PLEG): container finished" podID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerID="1d6fb68ceaffc0f23f79ae294c1490f040e36bda9093c449dd0596e28864d72d" exitCode=0 Oct 30 05:53:53.628190 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.628172 6999 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 05:53:53.633940 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.633920 6999 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:53:53.640306 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.640281 6999 generic.go:356] "Generic (PLEG): container finished" podID="a1192599-1c2d-4670-a19f-c4e9faa70c46" containerID="1032be3ba6dc8255623bd23eaf6d7dff303074262b9fd71a3bcf5518acdc2490" exitCode=0 Oct 30 05:53:53.655443 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.655420 6999 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:53:53.655608 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.655497 6999 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:53:53.655725 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.655634 6999 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:53:53.667619 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.667596 6999 generic.go:356] "Generic (PLEG): container finished" podID="e4a59e77-1b37-405f-b787-df3e7e2ecc43" containerID="c14da276786db7ef8e7f0d631410f083b423eff2d6fd0b5c4236056114253514" exitCode=0 Oct 30 05:53:53.667619 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.667617 6999 generic.go:356] "Generic (PLEG): container finished" podID="e4a59e77-1b37-405f-b787-df3e7e2ecc43" containerID="b96b894fcbc47bab7ffe04004e4199cf75f255255ad282f4240d2c147c4bb312" exitCode=0 Oct 30 05:53:53.667747 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.667623 6999 generic.go:356] "Generic (PLEG): container finished" podID="e4a59e77-1b37-405f-b787-df3e7e2ecc43" containerID="3269e40fbd2d6a16e95e5eda27a89e91b36456e35d61eed5f3acea181f7a65ab" exitCode=0 Oct 30 05:53:53.667747 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.667629 6999 generic.go:356] "Generic (PLEG): container finished" podID="e4a59e77-1b37-405f-b787-df3e7e2ecc43" containerID="6e27ea621e3ee4bb978efed557cd158630876f808b7cd793e1df368774628b27" exitCode=0 Oct 30 05:53:53.667747 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.667634 6999 generic.go:356] "Generic (PLEG): container finished" podID="e4a59e77-1b37-405f-b787-df3e7e2ecc43" containerID="643547053d5eba4a4debe8d85639c56b0446fd4ceeb430bbbf5855276a708613" exitCode=0 Oct 30 05:53:53.667747 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.667639 6999 generic.go:356] "Generic (PLEG): container finished" podID="e4a59e77-1b37-405f-b787-df3e7e2ecc43" containerID="4b40b3fdac93e95a3ca78718321d91f60b1308e252c43a8e8248c67e5f471db1" exitCode=0 Oct 30 05:53:53.673703 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.673685 6999 generic.go:356] "Generic (PLEG): container finished" podID="18e4d93490af52d80822ebfbfe232c01" containerID="4cb029c4d74de9c2c479a9de9b40cc3b203c48fb8c5bcb0ec406649f82d4ddb7" exitCode=0 Oct 30 05:53:53.700835 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:53.700788 6999 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 05:53:53.701451 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.701438 6999 manager.go:324] Recovery completed Oct 30 05:53:53.728454 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.728435 6999 cpu_manager.go:226] "Starting" policy="none" Oct 30 05:53:53.728454 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.728452 6999 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 05:53:53.728580 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.728472 6999 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 05:53:53.728639 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.728623 6999 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 05:53:53.728682 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.728642 6999 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 05:53:53.728718 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.728685 6999 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 05:53:53.728718 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.728693 6999 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 05:53:53.728718 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.728699 6999 policy_none.go:50] "Start" Oct 30 05:53:53.728718 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.728709 6999 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 05:53:53.728816 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.728720 6999 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 05:53:53.728871 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.728862 6999 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 05:53:53.728900 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.728872 6999 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 05:53:53.728900 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.728878 6999 policy_none.go:44] "Start" Oct 30 05:53:53.733577 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.733551 6999 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 05:53:53.736138 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.736121 6999 manager.go:341] "Starting Device Plugin manager" Oct 30 05:53:53.736628 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:53.736147 6999 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 05:53:53.736628 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.736627 6999 server.go:87] "Starting device plugin registration server" Oct 30 05:53:53.736893 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.736879 6999 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 05:53:53.736973 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.736890 6999 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 05:53:53.737026 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.736977 6999 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 05:53:53.737112 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.737083 6999 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 05:53:53.737162 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.737121 6999 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 05:53:53.737162 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.737127 6999 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 05:53:53.737281 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.737169 6999 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:53:53.737109955Z","UUID":"44fe22e1-467f-4c53-9c1f-bd9c0071e5af","Handler":null,"Name":"","Endpoint":""} Oct 30 05:53:53.738800 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.738534 6999 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 05:53:53.738800 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.738570 6999 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 05:53:53.739053 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:53.739018 6999 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 05:53:53.837557 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.837533 6999 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:53:53.839702 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.839684 6999 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:53:53.839787 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.839710 6999 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:53:53.839787 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.839719 6999 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-226.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:53:53.839787 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.839763 6999 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-226.ec2.internal" Oct 30 05:53:53.882846 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.882830 6999 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-226.ec2.internal" Oct 30 05:53:53.882904 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.882878 6999 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-226.ec2.internal" Oct 30 05:53:53.901884 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.901850 6999 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal"] Oct 30 05:53:53.902501 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.902484 6999 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:53:53.902610 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.902576 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" event={"ID":"0b8548329cbac7d7fb7409929225464e","Type":"ContainerStarted","Data":"fd40ada5509e946891ea0e4f3dbcc8c07304aa2ec77ff0690f9e5cb7235a8901"} Oct 30 05:53:53.902677 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.902625 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" event={"ID":"0b8548329cbac7d7fb7409929225464e","Type":"ContainerStarted","Data":"3b9d2a677fc04337562bb547eb666046282b130b11a94d933c41537e53849fef"} Oct 30 05:53:53.902743 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.902729 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" event={"ID":"18e4d93490af52d80822ebfbfe232c01","Type":"ContainerStarted","Data":"d2e85163fecdb49037614b5043abb3f73fc01dd68863dd945feaa0ccea8fd22d"} Oct 30 05:53:53.902790 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.902751 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" event={"ID":"18e4d93490af52d80822ebfbfe232c01","Type":"ContainerDied","Data":"4cb029c4d74de9c2c479a9de9b40cc3b203c48fb8c5bcb0ec406649f82d4ddb7"} Oct 30 05:53:53.902790 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.902763 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" event={"ID":"18e4d93490af52d80822ebfbfe232c01","Type":"ContainerStarted","Data":"8e0e5c1aa27c35c585015e70714593aa7aee54a1ff842315c94d39062be850f9"} Oct 30 05:53:53.902845 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.902796 6999 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" Oct 30 05:53:53.943302 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.943283 6999 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:53:53.943383 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:53.943325 6999 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-226.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" Oct 30 05:53:53.944976 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:53.944964 6999 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:53:53.945031 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:53.944991 6999 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:53:54.029342 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.029288 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b8548329cbac7d7fb7409929225464e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-226.ec2.internal\" (UID: \"0b8548329cbac7d7fb7409929225464e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" Oct 30 05:53:54.029342 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.029315 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18e4d93490af52d80822ebfbfe232c01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal\" (UID: \"18e4d93490af52d80822ebfbfe232c01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:53:54.029342 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.029331 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18e4d93490af52d80822ebfbfe232c01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal\" (UID: \"18e4d93490af52d80822ebfbfe232c01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:53:54.129838 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.129817 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18e4d93490af52d80822ebfbfe232c01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal\" (UID: \"18e4d93490af52d80822ebfbfe232c01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:53:54.129902 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.129842 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18e4d93490af52d80822ebfbfe232c01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal\" (UID: \"18e4d93490af52d80822ebfbfe232c01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:53:54.129902 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.129890 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b8548329cbac7d7fb7409929225464e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-226.ec2.internal\" (UID: \"0b8548329cbac7d7fb7409929225464e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" Oct 30 05:53:54.129980 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.129901 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b8548329cbac7d7fb7409929225464e-config\") pod \"kube-apiserver-proxy-ip-10-0-131-226.ec2.internal\" (UID: \"0b8548329cbac7d7fb7409929225464e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" Oct 30 05:53:54.129980 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.129928 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/18e4d93490af52d80822ebfbfe232c01-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal\" (UID: \"18e4d93490af52d80822ebfbfe232c01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:53:54.129980 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.129949 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/18e4d93490af52d80822ebfbfe232c01-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal\" (UID: \"18e4d93490af52d80822ebfbfe232c01\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:53:54.554604 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.554578 6999 apiserver.go:52] "Watching apiserver" Oct 30 05:53:54.619693 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.619673 6999 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 05:53:54.620397 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.620373 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal","openshift-network-operator/iptables-alerter-xjmhv","openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f","openshift-network-diagnostics/network-check-target-4gz57","openshift-ovn-kubernetes/ovnkube-node-dnbrk","kube-system/konnectivity-agent-8mvtx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x","openshift-monitoring/alertmanager-main-0","openshift-image-registry/node-ca-vjrj8","openshift-insights/insights-runtime-extractor-j7wwc","openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf","openshift-multus/multus-zfbpj","openshift-console/console-66db8fc9d4-pr2ll","openshift-dns/dns-default-tvz4j","openshift-ingress-canary/ingress-canary-fxbcf","openshift-cluster-node-tuning-operator/tuned-d29qr","kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal","openshift-dns/node-resolver-zqrdw","openshift-multus/network-metrics-daemon-kzpqd","openshift-monitoring/metrics-server-755798cb46-949x8","openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh","kube-system/global-pull-secret-syncer-pp7mt","openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7","openshift-multus/multus-additional-cni-plugins-nbcdv","openshift-monitoring/node-exporter-gfv67","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/telemeter-client-7ff58fc747-xt48n"] Oct 30 05:53:54.644820 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.644800 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 05:53:54.645183 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.645166 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 05:53:54.645251 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.645184 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 05:53:54.648345 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.648326 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 05:53:54.648446 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.648385 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 05:53:54.648446 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.648436 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 05:53:54.648801 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.648781 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 05:53:54.648864 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.648816 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:53:54.648864 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.648819 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:53:54.649086 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.649072 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:53:54.659336 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.659318 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-99gsm\"" Oct 30 05:53:54.668842 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.668824 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 05:53:54.668928 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.668852 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:53:54.671277 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.671254 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 05:53:54.671378 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.671321 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 05:53:54.671378 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.671331 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 05:53:54.671378 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.671355 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 05:53:54.671537 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.671411 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 05:53:54.671674 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.671646 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 05:53:54.671674 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.671656 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 05:53:54.671782 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.671749 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 05:53:54.671833 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.671823 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 05:53:54.671884 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.671826 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 05:53:54.671884 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.671810 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 05:53:54.671884 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.671824 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 05:53:54.672394 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.672374 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 05:53:54.672519 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.672397 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 05:53:54.672637 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.672435 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 05:53:54.672637 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.672545 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 05:53:54.672637 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.672536 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 05:53:54.672780 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.672751 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 05:53:54.673124 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.673073 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 05:53:54.674023 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.673994 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 05:53:54.674106 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.674052 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 05:53:54.675322 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.674828 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4fvwv\"" Oct 30 05:53:54.675322 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.675141 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 05:53:54.675322 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.675163 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 05:53:54.675322 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.675237 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 05:53:54.675322 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.675280 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 05:53:54.675661 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.675636 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q2ncw\"" Oct 30 05:53:54.675788 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.675770 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 05:53:54.675843 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.675806 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 05:53:54.675979 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.675962 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 05:53:54.676071 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.675812 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5x8x2\"" Oct 30 05:53:54.676071 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.676053 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 05:53:54.676264 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.676246 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 05:53:54.676333 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.676267 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 05:53:54.676448 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.676433 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 05:53:54.676538 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.676521 6999 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 05:53:54.676749 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.676733 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 05:53:54.677082 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.677056 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 05:53:54.677370 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.677350 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 05:53:54.677510 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.677477 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-xln8v\"" Oct 30 05:53:54.677891 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.677875 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-d9dv5\"" Oct 30 05:53:54.677959 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.677922 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:53:54.678072 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.678052 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 05:53:54.678272 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.678253 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 05:53:54.678346 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.678287 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lsk6t\"" Oct 30 05:53:54.678398 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.678375 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-72nb9\"" Oct 30 05:53:54.678448 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.678432 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-brovso9nv32hm\"" Oct 30 05:53:54.678537 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.678523 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 05:53:54.678632 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.678569 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 05:53:54.678632 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.678611 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 05:53:54.678632 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.678626 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 05:53:54.679382 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.679368 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 05:53:54.679609 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.679584 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-plsgv\"" Oct 30 05:53:54.679688 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.679606 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 05:53:54.679688 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.679636 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 05:53:54.679783 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.679761 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 05:53:54.679957 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.679927 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 05:53:54.680027 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.680011 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fkdst1vuu8j4a\"" Oct 30 05:53:54.680073 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.680036 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-27cwd\"" Oct 30 05:53:54.680117 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.680094 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-6hg99\"" Oct 30 05:53:54.680423 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.680406 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vzk7w\"" Oct 30 05:53:54.683558 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.683539 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-mkzdc\"" Oct 30 05:53:54.683661 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.683646 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 05:53:54.683725 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.683711 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rkzps\"" Oct 30 05:53:54.683814 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.683798 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 05:53:54.683857 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.683845 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 05:53:54.683971 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.683956 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 05:53:54.684027 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.683978 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5hkvn\"" Oct 30 05:53:54.684027 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.684014 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:53:54.684113 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.684097 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-x8h66\"" Oct 30 05:53:54.684156 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.684128 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-q46lp\"" Oct 30 05:53:54.685156 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.685142 6999 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" Oct 30 05:53:54.685321 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.685308 6999 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:53:54.686035 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.686014 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 05:53:54.686035 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.686030 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 05:53:54.686282 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.686265 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 05:53:54.686355 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.686338 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-kmt7k\"" Oct 30 05:53:54.686401 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.686336 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 05:53:54.686522 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.686506 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ntdkp\"" Oct 30 05:53:54.686591 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.686554 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 05:53:54.686591 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.686574 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 05:53:54.686591 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.686582 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 05:53:54.686704 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.686680 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 05:53:54.686741 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.686731 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r6cnz\"" Oct 30 05:53:54.686940 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.686929 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mpzcb\"" Oct 30 05:53:54.687265 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.687239 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2wt58\"" Oct 30 05:53:54.687372 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.687286 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 05:53:54.687372 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.687358 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-l4lh5\"" Oct 30 05:53:54.687919 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.687902 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 05:53:54.687992 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.687947 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 05:53:54.688305 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.688282 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 05:53:54.688305 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.688286 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 05:53:54.688305 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.688286 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 05:53:54.688403 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.688365 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 05:53:54.692529 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.692514 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 05:53:54.725072 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.725056 6999 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:53:54.725163 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.725102 6999 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-226.ec2.internal" Oct 30 05:53:54.726832 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.726818 6999 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:53:54.726948 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.726850 6999 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-226.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-226.ec2.internal" Oct 30 05:53:54.732185 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732168 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-secret-metrics-server-client-certs\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:54.732280 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732199 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcqpr\" (UniqueName: \"kubernetes.io/projected/e4a59e77-1b37-405f-b787-df3e7e2ecc43-kube-api-access-dcqpr\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.732280 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732244 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.732381 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732285 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:54.732381 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732318 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-system-cni-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.732381 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732344 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.732527 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732398 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.732527 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732433 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.732527 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732452 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:54.732527 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732470 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.732527 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732487 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-secret-metrics-server-client-certs\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:54.732527 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732495 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-k8s-cni-cncf-io\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.732527 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732520 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2g4l\" (UniqueName: \"kubernetes.io/projected/19c2a004-564f-442c-a29a-ebcd20f71102-kube-api-access-t2g4l\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732545 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8b944931-8f1e-4596-a265-ceec8c673872-metrics-server-audit-profiles\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732559 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732570 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.732576 6999 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732588 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732596 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-config-volume\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732622 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.732650 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:55.232629734 +0000 UTC m=+1.741649620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-oauth-config" not registered Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732678 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732753 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732787 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovnkube-script-lib\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732795 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732815 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-root\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732836 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-config-volume\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732844 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-conf-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.732858 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732865 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/8b944931-8f1e-4596-a265-ceec8c673872-metrics-server-audit-profiles\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:54.733469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732880 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.733469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732920 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.733469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732957 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.733469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.732981 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.733469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733038 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovnkube-script-lib\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.733469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733085 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv4cr\" (UniqueName: \"kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr\") pod \"network-check-target-4gz57\" (UID: \"3d693b0d-6752-422d-85d8-e8afb5a111b9\") " pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:54.733469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733118 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-system-cni-dir\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.733469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733164 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.733469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733264 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-ovn\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.733469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733300 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdl4j\" (UniqueName: \"kubernetes.io/projected/1b9d8b00-b8a3-4082-ad89-053a60ff7078-kube-api-access-zdl4j\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:53:54.733469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733322 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xggd\" (UniqueName: \"kubernetes.io/projected/0d76cf54-ecf2-4444-b7a1-024dcb3d215e-kube-api-access-6xggd\") pod \"ingress-canary-fxbcf\" (UID: \"0d76cf54-ecf2-4444-b7a1-024dcb3d215e\") " pod="openshift-ingress-canary/ingress-canary-fxbcf" Oct 30 05:53:54.733469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733418 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19c2a004-564f-442c-a29a-ebcd20f71102-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:54.733935 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733475 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-secret-metrics-server-tls\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:54.733935 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733571 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19c2a004-564f-442c-a29a-ebcd20f71102-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:54.733935 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733589 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b944931-8f1e-4596-a265-ceec8c673872-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:54.733935 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733621 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.733935 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733646 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q554\" (UniqueName: \"kubernetes.io/projected/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-kube-api-access-9q554\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:54.733935 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733708 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-secret-metrics-server-tls\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:54.733935 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733740 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.733935 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733748 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-wtmp\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.733935 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733789 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b944931-8f1e-4596-a265-ceec8c673872-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:54.733935 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733808 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:54.733935 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733837 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhpvd\" (UniqueName: \"kubernetes.io/projected/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-kube-api-access-lhpvd\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.733935 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733888 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.733935 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733908 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.733935 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733933 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733952 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-tls\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733970 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-kubelet\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.733993 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/733fbaf7-e75e-46c7-ad59-478b58424087-serviceca\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734021 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvn5s\" (UniqueName: \"kubernetes.io/projected/733fbaf7-e75e-46c7-ad59-478b58424087-kube-api-access-rvn5s\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734041 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734064 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpt2s\" (UniqueName: \"kubernetes.io/projected/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-kube-api-access-tpt2s\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734091 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-run\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734135 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e1f5208-4203-46bf-ab5f-8587b9be50de-tmp\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734140 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734159 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-grpc-tls\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734157 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734198 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-tls\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734203 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqk2s\" (UniqueName: \"kubernetes.io/projected/a1192599-1c2d-4670-a19f-c4e9faa70c46-kube-api-access-sqk2s\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734256 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9cb92418-7659-4dad-9ffc-fa4e29f099a4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734266 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e1f5208-4203-46bf-ab5f-8587b9be50de-tmp\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734288 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-out\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734285 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734370 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9cb92418-7659-4dad-9ffc-fa4e29f099a4-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734402 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-out\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734400 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-telemeter-client-tls\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734437 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-grpc-tls\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734463 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msst4\" (UniqueName: \"kubernetes.io/projected/e1db1c23-703c-4455-8bbf-23688bec2ede-kube-api-access-msst4\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734491 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734514 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734538 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-systemd-units\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734553 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734569 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysctl-conf\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.734578 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734586 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-lib-modules\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734630 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8b944931-8f1e-4596-a265-ceec8c673872-audit-log\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.734650 6999 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734682 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734686 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/733fbaf7-e75e-46c7-ad59-478b58424087-serviceca\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734694 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734717 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-host\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734749 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/8b944931-8f1e-4596-a265-ceec8c673872-audit-log\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.734768 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:55.234753909 +0000 UTC m=+1.743773799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"service-ca" not registered Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734791 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnx8\" (UniqueName: \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-kube-api-access-2jnx8\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734815 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734884 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgcmh\" (UniqueName: \"kubernetes.io/projected/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-api-access-hgcmh\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734891 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734912 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-multus-certs\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734941 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-host-slash\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734965 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-tls\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.734993 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-run-netns\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735061 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4a82cf91-ad7d-4d8f-b20b-00113772357b-crio-socket\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735117 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/19c2a004-564f-442c-a29a-ebcd20f71102-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735163 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysctl-d\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735189 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-tls\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735195 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735198 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-serving-certs-ca-bundle\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735306 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735325 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735342 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-systemd\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735358 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33bd8834-ad74-4370-b805-a6995dd16230-konnectivity-ca\") pod \"konnectivity-agent-8mvtx\" (UID: \"33bd8834-ad74-4370-b805-a6995dd16230\") " pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735380 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-etc-selinux\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735393 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/19c2a004-564f-442c-a29a-ebcd20f71102-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735405 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-serving-certs-ca-bundle\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735410 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7nmf\" (UniqueName: \"kubernetes.io/projected/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-kube-api-access-b7nmf\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735454 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config-out\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735562 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hng9h\" (UniqueName: \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-kube-api-access-hng9h\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735573 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33bd8834-ad74-4370-b805-a6995dd16230-konnectivity-ca\") pod \"konnectivity-agent-8mvtx\" (UID: \"33bd8834-ad74-4370-b805-a6995dd16230\") " pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735593 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config-out\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735629 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735635 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp89s\" (UniqueName: \"kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:54.735652 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735660 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735675 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-tmp\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735729 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-tmp\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735772 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735799 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mvkm\" (UniqueName: \"kubernetes.io/projected/12c1aa54-bfc9-4320-915c-59d1701de6d9-kube-api-access-9mvkm\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735815 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7mvt\" (UniqueName: \"kubernetes.io/projected/8b944931-8f1e-4596-a265-ceec8c673872-kube-api-access-f7mvt\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735833 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-kubelet\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735853 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735890 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a968117a-0683-4844-8936-99ef536894d9-kubelet-config\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735910 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735940 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735947 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735958 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrwht\" (UniqueName: \"kubernetes.io/projected/e2a06249-236e-43d7-b485-0d21b72fb442-kube-api-access-rrwht\") pod \"monitoring-plugin-77f4f76594-rzpvh\" (UID: \"e2a06249-236e-43d7-b485-0d21b72fb442\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735958 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a968117a-0683-4844-8936-99ef536894d9-original-pull-secret\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.735987 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-etc-kubernetes\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736003 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-iptables-alerter-script\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736091 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-sys\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736132 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736155 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-kubernetes\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736133 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736176 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736179 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736230 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-socket-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736270 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-registration-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736281 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-iptables-alerter-script\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736302 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-daemon-config\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736416 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736415 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e1f5208-4203-46bf-ab5f-8587b9be50de-metrics-client-ca\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736465 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44r7s\" (UniqueName: \"kubernetes.io/projected/4e1f5208-4203-46bf-ab5f-8587b9be50de-kube-api-access-44r7s\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736538 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-slash\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736545 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-daemon-config\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736560 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n69rf\" (UniqueName: \"kubernetes.io/projected/4a82cf91-ad7d-4d8f-b20b-00113772357b-kube-api-access-n69rf\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736582 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1192599-1c2d-4670-a19f-c4e9faa70c46-metrics-client-ca\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736596 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4e1f5208-4203-46bf-ab5f-8587b9be50de-metrics-client-ca\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736636 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736669 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736692 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1192599-1c2d-4670-a19f-c4e9faa70c46-metrics-client-ca\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736695 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-federate-client-tls\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736740 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-metrics-client-ca\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736744 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736769 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736802 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cb92418-7659-4dad-9ffc-fa4e29f099a4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736829 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-cnibin\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736849 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736854 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736901 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736945 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1db1c23-703c-4455-8bbf-23688bec2ede-metrics-client-ca\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736969 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-federate-client-tls\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.736977 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e2a06249-236e-43d7-b485-0d21b72fb442-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rzpvh\" (UID: \"e2a06249-236e-43d7-b485-0d21b72fb442\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737006 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d0e685-73c2-4a16-a54f-b6781627a9c8-tmp\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737007 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9cb92418-7659-4dad-9ffc-fa4e29f099a4-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737033 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-accelerators-collector-config\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737063 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-cni-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737064 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737095 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d0e685-73c2-4a16-a54f-b6781627a9c8-tmp\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737127 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-socket-dir-parent\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737146 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-etc-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737168 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4a82cf91-ad7d-4d8f-b20b-00113772357b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:54.737169 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737191 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-textfile\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737245 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e2a06249-236e-43d7-b485-0d21b72fb442-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rzpvh\" (UID: \"e2a06249-236e-43d7-b485-0d21b72fb442\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737252 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-accelerators-collector-config\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737257 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19c2a004-564f-442c-a29a-ebcd20f71102-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737268 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-textfile\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737288 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737307 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737331 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737342 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4a82cf91-ad7d-4d8f-b20b-00113772357b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737355 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737385 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-cni-netd\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737408 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovnkube-config\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.737414 6999 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737435 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-metrics-tls\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737442 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19c2a004-564f-442c-a29a-ebcd20f71102-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.737456 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:55.237445531 +0000 UTC m=+1.746465396 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737468 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33bd8834-ad74-4370-b805-a6995dd16230-agent-certs\") pod \"konnectivity-agent-8mvtx\" (UID: \"33bd8834-ad74-4370-b805-a6995dd16230\") " pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737503 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cnibin\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737608 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737621 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33bd8834-ad74-4370-b805-a6995dd16230-agent-certs\") pod \"konnectivity-agent-8mvtx\" (UID: \"33bd8834-ad74-4370-b805-a6995dd16230\") " pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737616 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cni-binary-copy\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737661 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-cni-bin\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737679 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-cni-bin\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737715 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-metrics-tls\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737736 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdfl5\" (UniqueName: \"kubernetes.io/projected/dd135f0d-de15-427c-b9f6-490eaf4a7540-kube-api-access-hdfl5\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737754 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovnkube-config\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737777 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtcl4\" (UniqueName: \"kubernetes.io/projected/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-kube-api-access-gtcl4\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737814 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cni-binary-copy\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737866 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d76cf54-ecf2-4444-b7a1-024dcb3d215e-cert\") pod \"ingress-canary-fxbcf\" (UID: \"0d76cf54-ecf2-4444-b7a1-024dcb3d215e\") " pod="openshift-ingress-canary/ingress-canary-fxbcf" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.737967 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738027 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-hostroot\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738102 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d76cf54-ecf2-4444-b7a1-024dcb3d215e-cert\") pod \"ingress-canary-fxbcf\" (UID: \"0d76cf54-ecf2-4444-b7a1-024dcb3d215e\") " pod="openshift-ingress-canary/ingress-canary-fxbcf" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738125 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738169 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovn-node-metrics-cert\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738190 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.738196 6999 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738242 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a968117a-0683-4844-8936-99ef536894d9-dbus\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738313 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9d8b00-b8a3-4082-ad89-053a60ff7078-tmp-dir\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.738340 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:55.2383244 +0000 UTC m=+1.747344265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-config" not registered Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738361 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4a82cf91-ad7d-4d8f-b20b-00113772357b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738386 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b9d8b00-b8a3-4082-ad89-053a60ff7078-tmp-dir\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738390 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-sys\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738415 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-systemd\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738434 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738440 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/12c1aa54-bfc9-4320-915c-59d1701de6d9-ovn-node-metrics-cert\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738449 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-web-config\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738469 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-node-log\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738505 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-env-overrides\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738541 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4a82cf91-ad7d-4d8f-b20b-00113772357b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738554 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-cni-binary-copy\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738664 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-netns\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738666 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/12c1aa54-bfc9-4320-915c-59d1701de6d9-env-overrides\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738768 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/733fbaf7-e75e-46c7-ad59-478b58424087-host\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738811 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-modprobe-d\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738831 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738828 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-cni-binary-copy\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738899 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-client-ca-bundle\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738930 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-log-socket\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738955 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b9d8b00-b8a3-4082-ad89-053a60ff7078-hosts-file\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738976 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/4e1f5208-4203-46bf-ab5f-8587b9be50de-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.738980 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-cni-multus\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739045 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739089 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-var-lib-kubelet\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739123 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-secret-telemeter-client\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739141 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/8b944931-8f1e-4596-a265-ceec8c673872-client-ca-bundle\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739243 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739263 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739276 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-metrics-certs\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739281 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-var-lib-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739309 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-device-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.739333 6999 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739352 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysconfig\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739380 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-volume\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.739469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739377 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-secret-telemeter-client\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.739677 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:55.239378089 +0000 UTC m=+1.748397971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739760 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739798 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739828 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739900 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-volume\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.739914 6999 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739935 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a82cf91-ad7d-4d8f-b20b-00113772357b-data-volume\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.739966 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-tmp-dir\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.739984 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:55.23996142 +0000 UTC m=+1.748981302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-serving-cert" not registered Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.740015 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-sys-fs\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.740016 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.740033 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-tmp-dir\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.740039 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4a82cf91-ad7d-4d8f-b20b-00113772357b-data-volume\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.740082 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-tuned\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.740266 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-web-config\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.740301 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.740344 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-tuned\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.740629 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.743054 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.740682 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-web-config\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.744099 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.743679 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 05:53:54.744832 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.744313 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.746152 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.746135 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 05:53:54.746455 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.746434 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 05:53:54.746536 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.746457 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 05:53:54.746536 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.746460 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 05:53:54.746536 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.746480 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 05:53:54.746669 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.746649 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 05:53:54.746669 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.746664 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9x2zl\"" Oct 30 05:53:54.746669 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.746649 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 05:53:54.746825 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.746659 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 05:53:54.746916 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.746901 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aqh338iohmgj7\"" Oct 30 05:53:54.746959 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.746924 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 05:53:54.747016 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.747003 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.747196 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.747184 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.748072 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.748051 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.748114 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.748078 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.749273 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.749258 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-web-config\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.750329 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.750315 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.750434 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.750421 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.753614 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.753589 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.754731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.754714 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1db1c23-703c-4455-8bbf-23688bec2ede-telemeter-client-tls\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.755837 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.755823 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.756951 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.756937 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.779756 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.779735 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2g4l\" (UniqueName: \"kubernetes.io/projected/19c2a004-564f-442c-a29a-ebcd20f71102-kube-api-access-t2g4l\") pod \"openshift-state-metrics-7bd7489957-hq6w7\" (UID: \"19c2a004-564f-442c-a29a-ebcd20f71102\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-hq6w7" Oct 30 05:53:54.780040 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.780026 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcqpr\" (UniqueName: \"kubernetes.io/projected/e4a59e77-1b37-405f-b787-df3e7e2ecc43-kube-api-access-dcqpr\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.780139 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.780125 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv4cr\" (UniqueName: \"kubernetes.io/projected/3d693b0d-6752-422d-85d8-e8afb5a111b9-kube-api-access-tv4cr\") pod \"network-check-target-4gz57\" (UID: \"3d693b0d-6752-422d-85d8-e8afb5a111b9\") " pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:53:54.781253 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.781234 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdl4j\" (UniqueName: \"kubernetes.io/projected/1b9d8b00-b8a3-4082-ad89-053a60ff7078-kube-api-access-zdl4j\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:53:54.804966 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.804926 6999 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 05:53:54.804966 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.804945 6999 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 05:53:54.804966 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.804955 6999 projected.go:196] Error preparing data for projected volume kube-api-access-cp89s for pod openshift-console/console-66db8fc9d4-pr2ll: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 05:53:54.805103 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:54.804999 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:55.304987746 +0000 UTC m=+1.814007654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cp89s" (UniqueName: "kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 05:53:54.805254 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.805238 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvn5s\" (UniqueName: \"kubernetes.io/projected/733fbaf7-e75e-46c7-ad59-478b58424087-kube-api-access-rvn5s\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:53:54.809552 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.809532 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44r7s\" (UniqueName: \"kubernetes.io/projected/4e1f5208-4203-46bf-ab5f-8587b9be50de-kube-api-access-44r7s\") pod \"thanos-querier-95d7f5dd9-d9f6f\" (UID: \"4e1f5208-4203-46bf-ab5f-8587b9be50de\") " pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:54.812441 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.812423 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtcl4\" (UniqueName: \"kubernetes.io/projected/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-kube-api-access-gtcl4\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.812553 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.812538 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpt2s\" (UniqueName: \"kubernetes.io/projected/f8fac18d-f521-4f23-ad5e-ec2b5dcac722-kube-api-access-tpt2s\") pod \"network-metrics-daemon-kzpqd\" (UID: \"f8fac18d-f521-4f23-ad5e-ec2b5dcac722\") " pod="openshift-multus/network-metrics-daemon-kzpqd" Oct 30 05:53:54.812611 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.812577 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q554\" (UniqueName: \"kubernetes.io/projected/3e9c238a-4ad9-4ad2-98d0-cb4509704ec0-kube-api-access-9q554\") pod \"dns-default-tvz4j\" (UID: \"3e9c238a-4ad9-4ad2-98d0-cb4509704ec0\") " pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:54.812775 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.812760 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrwht\" (UniqueName: \"kubernetes.io/projected/e2a06249-236e-43d7-b485-0d21b72fb442-kube-api-access-rrwht\") pod \"monitoring-plugin-77f4f76594-rzpvh\" (UID: \"e2a06249-236e-43d7-b485-0d21b72fb442\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" Oct 30 05:53:54.812807 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.812781 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7mvt\" (UniqueName: \"kubernetes.io/projected/8b944931-8f1e-4596-a265-ceec8c673872-kube-api-access-f7mvt\") pod \"metrics-server-755798cb46-949x8\" (UID: \"8b944931-8f1e-4596-a265-ceec8c673872\") " pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:53:54.813932 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.813917 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jnx8\" (UniqueName: \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-kube-api-access-2jnx8\") pod \"alertmanager-main-0\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:53:54.815035 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.815014 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhpvd\" (UniqueName: \"kubernetes.io/projected/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-kube-api-access-lhpvd\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.816866 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.816848 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdfl5\" (UniqueName: \"kubernetes.io/projected/dd135f0d-de15-427c-b9f6-490eaf4a7540-kube-api-access-hdfl5\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.816866 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.816861 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n69rf\" (UniqueName: \"kubernetes.io/projected/4a82cf91-ad7d-4d8f-b20b-00113772357b-kube-api-access-n69rf\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:54.817292 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.817273 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7nmf\" (UniqueName: \"kubernetes.io/projected/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-kube-api-access-b7nmf\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:53:54.822089 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.822072 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mvkm\" (UniqueName: \"kubernetes.io/projected/12c1aa54-bfc9-4320-915c-59d1701de6d9-kube-api-access-9mvkm\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.823995 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.823980 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hng9h\" (UniqueName: \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-kube-api-access-hng9h\") pod \"prometheus-k8s-0\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:54.825257 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.825243 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msst4\" (UniqueName: \"kubernetes.io/projected/e1db1c23-703c-4455-8bbf-23688bec2ede-kube-api-access-msst4\") pod \"telemeter-client-7ff58fc747-xt48n\" (UID: \"e1db1c23-703c-4455-8bbf-23688bec2ede\") " pod="openshift-monitoring/telemeter-client-7ff58fc747-xt48n" Oct 30 05:53:54.826543 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.826523 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqk2s\" (UniqueName: \"kubernetes.io/projected/a1192599-1c2d-4670-a19f-c4e9faa70c46-kube-api-access-sqk2s\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.829610 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.829595 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xggd\" (UniqueName: \"kubernetes.io/projected/0d76cf54-ecf2-4444-b7a1-024dcb3d215e-kube-api-access-6xggd\") pod \"ingress-canary-fxbcf\" (UID: \"0d76cf54-ecf2-4444-b7a1-024dcb3d215e\") " pod="openshift-ingress-canary/ingress-canary-fxbcf" Oct 30 05:53:54.830740 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.830720 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgcmh\" (UniqueName: \"kubernetes.io/projected/9cb92418-7659-4dad-9ffc-fa4e29f099a4-kube-api-access-hgcmh\") pod \"kube-state-metrics-5dc8d7d89b-5nppf\" (UID: \"9cb92418-7659-4dad-9ffc-fa4e29f099a4\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-5nppf" Oct 30 05:53:54.841627 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841607 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-slash\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.841683 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841633 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-cnibin\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.841683 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841649 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-cni-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.841683 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841671 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-socket-dir-parent\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.841790 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841705 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-etc-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.841790 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841720 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-socket-dir-parent\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.841790 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841741 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-slash\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.841790 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841748 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-cni-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.841790 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841763 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-cnibin\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.841790 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841761 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-cni-netd\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.841790 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841789 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-cni-netd\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841789 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-etc-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841796 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cnibin\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841815 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-cni-bin\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841830 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-cni-bin\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841852 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-cnibin\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841854 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-hostroot\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841880 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-cni-bin\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841890 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a968117a-0683-4844-8936-99ef536894d9-dbus\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841896 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-cni-bin\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841930 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-sys\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841940 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-hostroot\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841951 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-sys\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841953 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-systemd\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841943 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a968117a-0683-4844-8936-99ef536894d9-dbus\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841974 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-node-log\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.841989 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-netns\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842002 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/733fbaf7-e75e-46c7-ad59-478b58424087-host\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842007 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-systemd\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842024 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-modprobe-d\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842035 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-netns\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842041 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-node-log\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842055 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-log-socket\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.842059 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842055 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/733fbaf7-e75e-46c7-ad59-478b58424087-host\") pod \"node-ca-vjrj8\" (UID: \"733fbaf7-e75e-46c7-ad59-478b58424087\") " pod="openshift-image-registry/node-ca-vjrj8" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842078 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-log-socket\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842084 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b9d8b00-b8a3-4082-ad89-053a60ff7078-hosts-file\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842108 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-modprobe-d\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842111 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-cni-multus\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842140 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-var-lib-kubelet\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842144 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-cni-multus\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842157 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1b9d8b00-b8a3-4082-ad89-053a60ff7078-hosts-file\") pod \"node-resolver-zqrdw\" (UID: \"1b9d8b00-b8a3-4082-ad89-053a60ff7078\") " pod="openshift-dns/node-resolver-zqrdw" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842177 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-var-lib-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842195 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-device-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842228 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysconfig\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842244 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-var-lib-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842255 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-sys-fs\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842261 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-device-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842271 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842276 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-var-lib-kubelet\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842309 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-openvswitch\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842312 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysconfig\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842316 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-sys-fs\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842347 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-system-cni-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842388 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842407 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-system-cni-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842413 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-k8s-cni-cncf-io\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842435 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842441 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842465 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-root\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842492 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-conf-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842519 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842523 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-root\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842523 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-k8s-cni-cncf-io\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842537 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-system-cni-dir\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842532 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-multus-conf-dir\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842568 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e4a59e77-1b37-405f-b787-df3e7e2ecc43-system-cni-dir\") pod \"multus-additional-cni-plugins-nbcdv\" (UID: \"e4a59e77-1b37-405f-b787-df3e7e2ecc43\") " pod="openshift-multus/multus-additional-cni-plugins-nbcdv" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842582 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-ovn\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842602 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-wtmp\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842619 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-kubelet\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842630 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-ovn\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842633 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-run\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842654 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-run\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842679 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-var-lib-kubelet\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842685 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-systemd-units\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842705 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1192599-1c2d-4670-a19f-c4e9faa70c46-node-exporter-wtmp\") pod \"node-exporter-gfv67\" (UID: \"a1192599-1c2d-4670-a19f-c4e9faa70c46\") " pod="openshift-monitoring/node-exporter-gfv67" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842710 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842729 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-systemd-units\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842747 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysctl-conf\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842759 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-run-ovn-kubernetes\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842771 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-lib-modules\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842795 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-host\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842820 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-multus-certs\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842845 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-host-slash\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842848 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-host\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842852 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-host-run-multus-certs\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842874 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-run-netns\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842885 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-lib-modules\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842888 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b8ae8ad0-aec2-4513-a5b5-75e6246d92fb-host-slash\") pod \"iptables-alerter-xjmhv\" (UID: \"b8ae8ad0-aec2-4513-a5b5-75e6246d92fb\") " pod="openshift-network-operator/iptables-alerter-xjmhv" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842899 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-run-netns\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842879 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysctl-conf\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842901 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4a82cf91-ad7d-4d8f-b20b-00113772357b-crio-socket\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842940 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysctl-d\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842943 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4a82cf91-ad7d-4d8f-b20b-00113772357b-crio-socket\") pod \"insights-runtime-extractor-j7wwc\" (UID: \"4a82cf91-ad7d-4d8f-b20b-00113772357b\") " pod="openshift-insights/insights-runtime-extractor-j7wwc" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842957 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-systemd\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.842979 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-etc-selinux\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843004 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-run-systemd\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843016 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-kubelet\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843037 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-etc-selinux\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843037 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843061 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-kubelet\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843062 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-sysctl-d\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.843039 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843062 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a968117a-0683-4844-8936-99ef536894d9-kubelet-config\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:54.844848 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843096 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12c1aa54-bfc9-4320-915c-59d1701de6d9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dnbrk\" (UID: \"12c1aa54-bfc9-4320-915c-59d1701de6d9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:54.844848 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843112 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a968117a-0683-4844-8936-99ef536894d9-kubelet-config\") pod \"global-pull-secret-syncer-pp7mt\" (UID: \"a968117a-0683-4844-8936-99ef536894d9\") " pod="kube-system/global-pull-secret-syncer-pp7mt" Oct 30 05:53:54.844848 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843114 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-etc-kubernetes\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.844848 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843142 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-sys\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.844848 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843142 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26c62aa1-d87b-4997-ae6a-ed1957b6ea73-etc-kubernetes\") pod \"multus-zfbpj\" (UID: \"26c62aa1-d87b-4997-ae6a-ed1957b6ea73\") " pod="openshift-multus/multus-zfbpj" Oct 30 05:53:54.844848 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843174 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-kubernetes\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.844848 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843188 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-sys\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.844848 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843198 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-socket-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.844848 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843232 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-registration-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.844848 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843251 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aa4aa89-688f-4a17-b2bc-88dbcd856db8-etc-kubernetes\") pod \"tuned-d29qr\" (UID: \"6aa4aa89-688f-4a17-b2bc-88dbcd856db8\") " pod="openshift-cluster-node-tuning-operator/tuned-d29qr" Oct 30 05:53:54.844848 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843263 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-socket-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:54.844848 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:54.843280 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dd135f0d-de15-427c-b9f6-490eaf4a7540-registration-dir\") pod \"aws-ebs-csi-driver-node-6w55x\" (UID: \"dd135f0d-de15-427c-b9f6-490eaf4a7540\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6w55x" Oct 30 05:53:55.216003 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:55.215976 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-95d7f5dd9-d9f6f" Oct 30 05:53:55.246249 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:55.246194 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:55.246378 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:55.246305 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:55.246378 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.246309 6999 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 05:53:55.246378 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:55.246331 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:55.246378 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.246357 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:56.246345524 +0000 UTC m=+2.755365390 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"service-ca" not registered Oct 30 05:53:55.246378 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.246378 6999 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 05:53:55.246619 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:55.246393 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:55.246619 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.246414 6999 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 05:53:55.246619 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:55.246436 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:55.246619 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.246461 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:56.246451118 +0000 UTC m=+2.755470983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-config" not registered Oct 30 05:53:55.246619 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.246485 6999 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 05:53:55.246619 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.246486 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:56.246476836 +0000 UTC m=+2.755496703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 05:53:55.246619 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.246516 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:56.246505548 +0000 UTC m=+2.755525415 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 05:53:55.246619 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.246522 6999 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 05:53:55.246619 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:55.246546 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:55.246619 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.246555 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:56.246545421 +0000 UTC m=+2.755565306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-serving-cert" not registered Oct 30 05:53:55.246619 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.246600 6999 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 05:53:55.247018 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.246638 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:56.246625202 +0000 UTC m=+2.755645070 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-oauth-config" not registered Oct 30 05:53:55.347708 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:55.347681 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp89s\" (UniqueName: \"kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:55.347856 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.347839 6999 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 05:53:55.347922 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.347862 6999 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 05:53:55.347922 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.347873 6999 projected.go:196] Error preparing data for projected volume kube-api-access-cp89s for pod openshift-console/console-66db8fc9d4-pr2ll: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 05:53:55.347998 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:55.347934 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:56.347918361 +0000 UTC m=+2.856938245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cp89s" (UniqueName: "kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 05:53:55.453305 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:55.453274 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:53:56.183596 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:56.183560 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:56.256878 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:56.256847 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:56.257022 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:56.256891 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:56.257022 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:56.256948 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:56.257022 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:56.256968 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:56.257022 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:56.256999 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:56.257022 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.257011 6999 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 05:53:56.257259 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.257047 6999 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 05:53:56.257259 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:56.257053 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:56.257259 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.257070 6999 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 05:53:56.257259 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.257081 6999 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 05:53:56.257259 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.257047 6999 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 05:53:56.257259 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.257097 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:58.257076351 +0000 UTC m=+4.766096231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-oauth-config" not registered Oct 30 05:53:56.257259 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.257123 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:58.257112104 +0000 UTC m=+4.766131973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 05:53:56.257259 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.257131 6999 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 05:53:56.257259 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.257144 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:58.257131619 +0000 UTC m=+4.766151490 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"service-ca" not registered Oct 30 05:53:56.257259 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.257189 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:58.257178969 +0000 UTC m=+4.766198835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 05:53:56.257259 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.257204 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:58.257195875 +0000 UTC m=+4.766215743 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-config" not registered Oct 30 05:53:56.257259 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.257245 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:58.257209199 +0000 UTC m=+4.766229068 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-serving-cert" not registered Oct 30 05:53:56.290069 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:56.290049 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:56.358113 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:56.358085 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp89s\" (UniqueName: \"kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:56.358321 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.358245 6999 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 05:53:56.358321 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.358260 6999 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 05:53:56.358321 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.358269 6999 projected.go:196] Error preparing data for projected volume kube-api-access-cp89s for pod openshift-console/console-66db8fc9d4-pr2ll: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 05:53:56.358321 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:56.358307 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:53:58.358295834 +0000 UTC m=+4.867315702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cp89s" (UniqueName: "kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 05:53:56.395084 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:56.395064 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:53:56.886711 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:56.886677 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tvz4j" Oct 30 05:53:58.277465 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:58.277432 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:58.277474 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:58.277494 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:58.277524 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.277570 6999 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:58.277572 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.277606 6999 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.277606 6999 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.277571 6999 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.277653 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:02.277637339 +0000 UTC m=+8.786657204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"service-ca" not registered Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.277670 6999 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.277673 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:02.277663372 +0000 UTC m=+8.786683270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.277728 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:02.277716074 +0000 UTC m=+8.786735940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-serving-cert" not registered Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.277742 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:02.277733959 +0000 UTC m=+8.786753823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-config" not registered Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.277754 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:02.277749058 +0000 UTC m=+8.786768922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-oauth-config" not registered Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:58.277794 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.277835 6999 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 05:53:58.277874 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.277864 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:02.277857689 +0000 UTC m=+8.786877591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 05:53:58.378181 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:58.378159 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp89s\" (UniqueName: \"kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:53:58.378328 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.378304 6999 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 05:53:58.378328 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.378320 6999 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 05:53:58.378396 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.378332 6999 projected.go:196] Error preparing data for projected volume kube-api-access-cp89s for pod openshift-console/console-66db8fc9d4-pr2ll: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 05:53:58.378396 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:53:58.378372 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:02.378360489 +0000 UTC m=+8.887380354 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cp89s" (UniqueName: "kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 05:53:58.485570 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:58.485541 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:53:58.485739 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:58.485728 6999 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:53:59.413367 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:53:59.413311 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:54:00.707340 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:00.707308 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:54:00.870437 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:00.870392 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:54:02.318265 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:02.318204 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:02.318265 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:02.318274 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.318288 6999 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.318337 6999 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:02.318300 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.318360 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:10.318338683 +0000 UTC m=+16.827358547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.318379 6999 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.318395 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:10.318380578 +0000 UTC m=+16.827400444 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.318423 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:10.31841147 +0000 UTC m=+16.827431336 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-config" not registered Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:02.318447 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:02.318478 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:02.318524 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.318561 6999 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.318591 6999 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.318604 6999 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.318615 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:10.3186034 +0000 UTC m=+16.827623285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-serving-cert" not registered Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.318631 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:10.318622267 +0000 UTC m=+16.827642164 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-oauth-config" not registered Oct 30 05:54:02.318706 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.318643 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:10.318637376 +0000 UTC m=+16.827657241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"service-ca" not registered Oct 30 05:54:02.419586 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:02.419546 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp89s\" (UniqueName: \"kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:02.419729 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.419691 6999 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 05:54:02.419729 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.419705 6999 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 05:54:02.419729 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.419714 6999 projected.go:196] Error preparing data for projected volume kube-api-access-cp89s for pod openshift-console/console-66db8fc9d4-pr2ll: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 05:54:02.419828 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:02.419757 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:10.419746099 +0000 UTC m=+16.928765964 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cp89s" (UniqueName: "kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 05:54:02.781493 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:02.781462 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rzpvh" Oct 30 05:54:04.139451 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:04.139377 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 05:54:10.394326 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:10.394296 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:10.394334 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:10.394356 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:10.394386 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.394438 6999 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.394456 6999 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.394461 6999 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.394484 6999 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.394510 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:26.3944966 +0000 UTC m=+32.903516466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"service-ca" not registered Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.394530 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:26.394516255 +0000 UTC m=+32.903536119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-serving-cert" not registered Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.394544 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:26.394537969 +0000 UTC m=+32.903557834 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.394554 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:26.394548995 +0000 UTC m=+32.903568859 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-oauth-config" not registered Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:10.394573 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:10.394606 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.394671 6999 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.394673 6999 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.394711 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:26.394700058 +0000 UTC m=+32.903719937 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"trusted-ca-bundle" not registered Oct 30 05:54:10.394731 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.394724 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:26.394718451 +0000 UTC m=+32.903738315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : object "openshift-console"/"console-config" not registered Oct 30 05:54:10.495042 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:10.495013 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp89s\" (UniqueName: \"kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s\") pod \"console-66db8fc9d4-pr2ll\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:10.495202 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.495149 6999 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 05:54:10.495202 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.495163 6999 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 05:54:10.495202 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.495172 6999 projected.go:196] Error preparing data for projected volume kube-api-access-cp89s for pod openshift-console/console-66db8fc9d4-pr2ll: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 05:54:10.495320 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:10.495232 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s podName:19d0e685-73c2-4a16-a54f-b6781627a9c8 nodeName:}" failed. No retries permitted until 2026-10-30 05:54:26.495203631 +0000 UTC m=+33.004223496 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cp89s" (UniqueName: "kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s") pod "console-66db8fc9d4-pr2ll" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 05:54:16.889198 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:16.889163 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-755798cb46-949x8" Oct 30 05:54:18.223302 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:18.223273 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:54:18.383531 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:18.383489 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:54:19.367077 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:19.367038 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:54:19.511991 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:19.511961 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:54:19.655498 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:19.655427 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-66db8fc9d4-pr2ll" podUID="19d0e685-73c2-4a16-a54f-b6781627a9c8" containerName="console" containerID="cri-o://8426d98f2edeb3f6282cca309d46a9ebadc67a3fb93d696eb3ee6abdcd84f845" gracePeriod=15 Oct 30 05:54:19.661787 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:19.661762 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:54:19.778705 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:19.778682 6999 generic.go:356] "Generic (PLEG): container finished" podID="19d0e685-73c2-4a16-a54f-b6781627a9c8" containerID="8426d98f2edeb3f6282cca309d46a9ebadc67a3fb93d696eb3ee6abdcd84f845" exitCode=0 Oct 30 05:54:19.778816 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:19.778757 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66db8fc9d4-pr2ll" event={"ID":"19d0e685-73c2-4a16-a54f-b6781627a9c8","Type":"ContainerDied","Data":"8426d98f2edeb3f6282cca309d46a9ebadc67a3fb93d696eb3ee6abdcd84f845"} Oct 30 05:54:19.931675 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:19.931623 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:20.086058 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.086031 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config\") pod \"19d0e685-73c2-4a16-a54f-b6781627a9c8\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " Oct 30 05:54:20.086248 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.086071 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert\") pod \"19d0e685-73c2-4a16-a54f-b6781627a9c8\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " Oct 30 05:54:20.086248 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.086101 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert\") pod \"19d0e685-73c2-4a16-a54f-b6781627a9c8\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " Oct 30 05:54:20.086248 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.086127 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca\") pod \"19d0e685-73c2-4a16-a54f-b6781627a9c8\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " Oct 30 05:54:20.086248 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.086152 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config\") pod \"19d0e685-73c2-4a16-a54f-b6781627a9c8\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " Oct 30 05:54:20.086248 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.086182 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s\" (UniqueName: \"kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s\") pod \"19d0e685-73c2-4a16-a54f-b6781627a9c8\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " Oct 30 05:54:20.086520 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.086263 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle\") pod \"19d0e685-73c2-4a16-a54f-b6781627a9c8\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " Oct 30 05:54:20.086520 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.086354 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19d0e685-73c2-4a16-a54f-b6781627a9c8-tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d0e685-73c2-4a16-a54f-b6781627a9c8-tmp\") pod \"19d0e685-73c2-4a16-a54f-b6781627a9c8\" (UID: \"19d0e685-73c2-4a16-a54f-b6781627a9c8\") " Oct 30 05:54:20.086520 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.086446 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert" pod "19d0e685-73c2-4a16-a54f-b6781627a9c8" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:20.086678 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.086637 6999 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-oauth-serving-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:20.086728 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.086654 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d0e685-73c2-4a16-a54f-b6781627a9c8-tmp" pod "19d0e685-73c2-4a16-a54f-b6781627a9c8" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:54:20.086780 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.086747 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config" pod "19d0e685-73c2-4a16-a54f-b6781627a9c8" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:20.086834 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.086813 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle" pod "19d0e685-73c2-4a16-a54f-b6781627a9c8" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:20.087050 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.086982 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca" pod "19d0e685-73c2-4a16-a54f-b6781627a9c8" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:20.088627 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.088605 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config" pod "19d0e685-73c2-4a16-a54f-b6781627a9c8" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:20.088753 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.088726 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert" pod "19d0e685-73c2-4a16-a54f-b6781627a9c8" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:20.088753 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.088738 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s" pod "19d0e685-73c2-4a16-a54f-b6781627a9c8" (UID: "19d0e685-73c2-4a16-a54f-b6781627a9c8"). InnerVolumeSpecName "kube-api-access-cp89s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:20.187947 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.187896 6999 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-config\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:20.187947 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.187917 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp89s\" (UniqueName: \"kubernetes.io/projected/19d0e685-73c2-4a16-a54f-b6781627a9c8-kube-api-access-cp89s\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:20.187947 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.187927 6999 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-trusted-ca-bundle\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:20.187947 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.187936 6999 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d0e685-73c2-4a16-a54f-b6781627a9c8-tmp\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:20.187947 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.187944 6999 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-oauth-config\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:20.188154 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.187954 6999 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19d0e685-73c2-4a16-a54f-b6781627a9c8-console-serving-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:20.188154 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.187962 6999 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19d0e685-73c2-4a16-a54f-b6781627a9c8-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:20.783817 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.783785 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66db8fc9d4-pr2ll" event={"ID":"19d0e685-73c2-4a16-a54f-b6781627a9c8","Type":"ContainerDied","Data":"7e97c158e1a2368bd2d8d8113b8eece430e9677d11d4e3c574161f1a8abe121b"} Oct 30 05:54:20.784250 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.783825 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66db8fc9d4-pr2ll" Oct 30 05:54:20.784250 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.783840 6999 scope.go:122] "RemoveContainer" containerID="8426d98f2edeb3f6282cca309d46a9ebadc67a3fb93d696eb3ee6abdcd84f845" Oct 30 05:54:20.839571 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.839542 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66db8fc9d4-pr2ll"] Oct 30 05:54:20.852552 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:20.852527 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66db8fc9d4-pr2ll"] Oct 30 05:54:21.606036 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:21.606005 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19d0e685-73c2-4a16-a54f-b6781627a9c8" path="/var/lib/kubelet/pods/19d0e685-73c2-4a16-a54f-b6781627a9c8/volumes" Oct 30 05:54:23.342117 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.342079 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:54:23.342894 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.342861 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="alertmanager" containerID="cri-o://dc08fbe805e214511e7b406f3e6f4583f0a548015c82503731efdb53690fa7ab" gracePeriod=120 Oct 30 05:54:23.343620 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.343187 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="kube-rbac-proxy" containerID="cri-o://52e85192fd0abb256a2e2c693dcec734355a33580f71e1435f0fe3a128e3ea10" gracePeriod=120 Oct 30 05:54:23.343620 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.343287 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="kube-rbac-proxy-web" containerID="cri-o://107e86592e5a37166c43840d3f66816b76736d29010ed3f147b5b753dffd7c71" gracePeriod=120 Oct 30 05:54:23.343620 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.343370 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="prom-label-proxy" containerID="cri-o://1095cbe0c9e5b3d06752eab58eab158f24d0059ed18bd84bccdd5b2bb1f01ebb" gracePeriod=120 Oct 30 05:54:23.343620 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.343422 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="config-reloader" containerID="cri-o://8e344635d04e78d8822773fe6d87d5e8bb56e101731179354bb2b7c7743143ff" gracePeriod=120 Oct 30 05:54:23.343620 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.343465 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="kube-rbac-proxy-metric" containerID="cri-o://c8e6a5406deb9dd647b72e5013d778ddb66f11eed587beb381249125365a61bd" gracePeriod=120 Oct 30 05:54:23.799476 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.799446 6999 generic.go:356] "Generic (PLEG): container finished" podID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerID="1095cbe0c9e5b3d06752eab58eab158f24d0059ed18bd84bccdd5b2bb1f01ebb" exitCode=0 Oct 30 05:54:23.799476 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.799471 6999 generic.go:356] "Generic (PLEG): container finished" podID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerID="c8e6a5406deb9dd647b72e5013d778ddb66f11eed587beb381249125365a61bd" exitCode=0 Oct 30 05:54:23.799476 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.799477 6999 generic.go:356] "Generic (PLEG): container finished" podID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerID="52e85192fd0abb256a2e2c693dcec734355a33580f71e1435f0fe3a128e3ea10" exitCode=0 Oct 30 05:54:23.799476 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.799482 6999 generic.go:356] "Generic (PLEG): container finished" podID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerID="8e344635d04e78d8822773fe6d87d5e8bb56e101731179354bb2b7c7743143ff" exitCode=0 Oct 30 05:54:23.799727 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.799487 6999 generic.go:356] "Generic (PLEG): container finished" podID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerID="dc08fbe805e214511e7b406f3e6f4583f0a548015c82503731efdb53690fa7ab" exitCode=0 Oct 30 05:54:23.799727 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.799506 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b21885d3-9120-4c95-81ad-cdb80fcdf022","Type":"ContainerDied","Data":"1095cbe0c9e5b3d06752eab58eab158f24d0059ed18bd84bccdd5b2bb1f01ebb"} Oct 30 05:54:23.799727 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.799570 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b21885d3-9120-4c95-81ad-cdb80fcdf022","Type":"ContainerDied","Data":"c8e6a5406deb9dd647b72e5013d778ddb66f11eed587beb381249125365a61bd"} Oct 30 05:54:23.799727 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.799596 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b21885d3-9120-4c95-81ad-cdb80fcdf022","Type":"ContainerDied","Data":"52e85192fd0abb256a2e2c693dcec734355a33580f71e1435f0fe3a128e3ea10"} Oct 30 05:54:23.799727 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.799618 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b21885d3-9120-4c95-81ad-cdb80fcdf022","Type":"ContainerDied","Data":"8e344635d04e78d8822773fe6d87d5e8bb56e101731179354bb2b7c7743143ff"} Oct 30 05:54:23.799727 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:23.799640 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b21885d3-9120-4c95-81ad-cdb80fcdf022","Type":"ContainerDied","Data":"dc08fbe805e214511e7b406f3e6f4583f0a548015c82503731efdb53690fa7ab"} Oct 30 05:54:24.268525 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.268496 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8mvtx" Oct 30 05:54:24.514609 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.514584 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dnbrk" Oct 30 05:54:24.582225 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.582196 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:24.726277 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.726246 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy\") pod \"b21885d3-9120-4c95-81ad-cdb80fcdf022\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " Oct 30 05:54:24.726459 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.726303 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-tls-assets\" (UniqueName: \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-tls-assets\") pod \"b21885d3-9120-4c95-81ad-cdb80fcdf022\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " Oct 30 05:54:24.726459 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.726345 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-cluster-tls-config\") pod \"b21885d3-9120-4c95-81ad-cdb80fcdf022\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " Oct 30 05:54:24.726459 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.726389 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-metrics-client-ca\") pod \"b21885d3-9120-4c95-81ad-cdb80fcdf022\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " Oct 30 05:54:24.726459 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.726427 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b21885d3-9120-4c95-81ad-cdb80fcdf022\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " Oct 30 05:54:24.726459 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.726455 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-web-config\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-web-config\") pod \"b21885d3-9120-4c95-81ad-cdb80fcdf022\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " Oct 30 05:54:24.726725 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.726480 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-main-tls\") pod \"b21885d3-9120-4c95-81ad-cdb80fcdf022\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " Oct 30 05:54:24.726725 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.726518 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-web\") pod \"b21885d3-9120-4c95-81ad-cdb80fcdf022\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " Oct 30 05:54:24.726725 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.726549 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-kube-api-access-2jnx8\" (UniqueName: \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-kube-api-access-2jnx8\") pod \"b21885d3-9120-4c95-81ad-cdb80fcdf022\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " Oct 30 05:54:24.726725 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.726592 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-main-db\") pod \"b21885d3-9120-4c95-81ad-cdb80fcdf022\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " Oct 30 05:54:24.726725 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.726622 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-volume\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-volume\") pod \"b21885d3-9120-4c95-81ad-cdb80fcdf022\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " Oct 30 05:54:24.726725 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.726655 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-trusted-ca-bundle\") pod \"b21885d3-9120-4c95-81ad-cdb80fcdf022\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " Oct 30 05:54:24.726725 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.726693 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-out\") pod \"b21885d3-9120-4c95-81ad-cdb80fcdf022\" (UID: \"b21885d3-9120-4c95-81ad-cdb80fcdf022\") " Oct 30 05:54:24.727117 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.727089 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-metrics-client-ca" pod "b21885d3-9120-4c95-81ad-cdb80fcdf022" (UID: "b21885d3-9120-4c95-81ad-cdb80fcdf022"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:24.729566 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.729533 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-main-db" pod "b21885d3-9120-4c95-81ad-cdb80fcdf022" (UID: "b21885d3-9120-4c95-81ad-cdb80fcdf022"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:54:24.729825 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.729805 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-out" pod "b21885d3-9120-4c95-81ad-cdb80fcdf022" (UID: "b21885d3-9120-4c95-81ad-cdb80fcdf022"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:54:24.731508 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.731482 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-metric" pod "b21885d3-9120-4c95-81ad-cdb80fcdf022" (UID: "b21885d3-9120-4c95-81ad-cdb80fcdf022"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:24.732018 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.731906 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy" pod "b21885d3-9120-4c95-81ad-cdb80fcdf022" (UID: "b21885d3-9120-4c95-81ad-cdb80fcdf022"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:24.732319 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.732293 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-tls-assets" pod "b21885d3-9120-4c95-81ad-cdb80fcdf022" (UID: "b21885d3-9120-4c95-81ad-cdb80fcdf022"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:24.732630 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.732547 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-main-tls" pod "b21885d3-9120-4c95-81ad-cdb80fcdf022" (UID: "b21885d3-9120-4c95-81ad-cdb80fcdf022"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:24.732630 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.732595 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-kube-api-access-2jnx8" pod "b21885d3-9120-4c95-81ad-cdb80fcdf022" (UID: "b21885d3-9120-4c95-81ad-cdb80fcdf022"). InnerVolumeSpecName "kube-api-access-2jnx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:24.732760 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.732730 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-volume" pod "b21885d3-9120-4c95-81ad-cdb80fcdf022" (UID: "b21885d3-9120-4c95-81ad-cdb80fcdf022"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:24.732821 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.732797 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-trusted-ca-bundle" pod "b21885d3-9120-4c95-81ad-cdb80fcdf022" (UID: "b21885d3-9120-4c95-81ad-cdb80fcdf022"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:24.733373 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.733347 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-web" pod "b21885d3-9120-4c95-81ad-cdb80fcdf022" (UID: "b21885d3-9120-4c95-81ad-cdb80fcdf022"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:24.735925 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.735810 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-cluster-tls-config" pod "b21885d3-9120-4c95-81ad-cdb80fcdf022" (UID: "b21885d3-9120-4c95-81ad-cdb80fcdf022"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:24.743623 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.743601 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-web-config" pod "b21885d3-9120-4c95-81ad-cdb80fcdf022" (UID: "b21885d3-9120-4c95-81ad-cdb80fcdf022"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:24.806093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.806067 6999 generic.go:356] "Generic (PLEG): container finished" podID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerID="107e86592e5a37166c43840d3f66816b76736d29010ed3f147b5b753dffd7c71" exitCode=0 Oct 30 05:54:24.806239 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.806147 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b21885d3-9120-4c95-81ad-cdb80fcdf022","Type":"ContainerDied","Data":"107e86592e5a37166c43840d3f66816b76736d29010ed3f147b5b753dffd7c71"} Oct 30 05:54:24.806239 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.806195 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:24.806318 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.806261 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b21885d3-9120-4c95-81ad-cdb80fcdf022","Type":"ContainerDied","Data":"2ac00632bbad142c4aa0561179ca2a76a346129f868db53b34119c6704e43339"} Oct 30 05:54:24.806318 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.806297 6999 scope.go:122] "RemoveContainer" containerID="1095cbe0c9e5b3d06752eab58eab158f24d0059ed18bd84bccdd5b2bb1f01ebb" Oct 30 05:54:24.813736 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.813718 6999 scope.go:122] "RemoveContainer" containerID="c8e6a5406deb9dd647b72e5013d778ddb66f11eed587beb381249125365a61bd" Oct 30 05:54:24.820742 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.820726 6999 scope.go:122] "RemoveContainer" containerID="52e85192fd0abb256a2e2c693dcec734355a33580f71e1435f0fe3a128e3ea10" Oct 30 05:54:24.827562 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.827538 6999 scope.go:122] "RemoveContainer" containerID="107e86592e5a37166c43840d3f66816b76736d29010ed3f147b5b753dffd7c71" Oct 30 05:54:24.827907 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.827885 6999 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:24.827968 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.827915 6999 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-tls-assets\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:24.827968 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.827932 6999 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-cluster-tls-config\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:24.827968 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.827947 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-metrics-client-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:24.827968 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.827961 6999 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:24.828112 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.827976 6999 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-web-config\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:24.828112 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.827990 6999 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-main-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:24.828112 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.828004 6999 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:24.828112 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.828018 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jnx8\" (UniqueName: \"kubernetes.io/projected/b21885d3-9120-4c95-81ad-cdb80fcdf022-kube-api-access-2jnx8\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:24.828112 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.828032 6999 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-main-db\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:24.828112 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.828046 6999 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:24.828112 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.828059 6999 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b21885d3-9120-4c95-81ad-cdb80fcdf022-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:24.828112 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.828072 6999 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b21885d3-9120-4c95-81ad-cdb80fcdf022-config-out\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:24.833972 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.833951 6999 scope.go:122] "RemoveContainer" containerID="8e344635d04e78d8822773fe6d87d5e8bb56e101731179354bb2b7c7743143ff" Oct 30 05:54:24.840852 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.840835 6999 scope.go:122] "RemoveContainer" containerID="dc08fbe805e214511e7b406f3e6f4583f0a548015c82503731efdb53690fa7ab" Oct 30 05:54:24.848595 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.848580 6999 scope.go:122] "RemoveContainer" containerID="1d6fb68ceaffc0f23f79ae294c1490f040e36bda9093c449dd0596e28864d72d" Oct 30 05:54:24.855350 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.855334 6999 scope.go:122] "RemoveContainer" containerID="1095cbe0c9e5b3d06752eab58eab158f24d0059ed18bd84bccdd5b2bb1f01ebb" Oct 30 05:54:24.855632 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:24.855604 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1095cbe0c9e5b3d06752eab58eab158f24d0059ed18bd84bccdd5b2bb1f01ebb\": container with ID starting with 1095cbe0c9e5b3d06752eab58eab158f24d0059ed18bd84bccdd5b2bb1f01ebb not found: ID does not exist" containerID="1095cbe0c9e5b3d06752eab58eab158f24d0059ed18bd84bccdd5b2bb1f01ebb" Oct 30 05:54:24.855690 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.855640 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1095cbe0c9e5b3d06752eab58eab158f24d0059ed18bd84bccdd5b2bb1f01ebb"} err="failed to get container status \"1095cbe0c9e5b3d06752eab58eab158f24d0059ed18bd84bccdd5b2bb1f01ebb\": rpc error: code = NotFound desc = could not find container \"1095cbe0c9e5b3d06752eab58eab158f24d0059ed18bd84bccdd5b2bb1f01ebb\": container with ID starting with 1095cbe0c9e5b3d06752eab58eab158f24d0059ed18bd84bccdd5b2bb1f01ebb not found: ID does not exist" Oct 30 05:54:24.855690 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.855671 6999 scope.go:122] "RemoveContainer" containerID="c8e6a5406deb9dd647b72e5013d778ddb66f11eed587beb381249125365a61bd" Oct 30 05:54:24.855925 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:24.855910 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8e6a5406deb9dd647b72e5013d778ddb66f11eed587beb381249125365a61bd\": container with ID starting with c8e6a5406deb9dd647b72e5013d778ddb66f11eed587beb381249125365a61bd not found: ID does not exist" containerID="c8e6a5406deb9dd647b72e5013d778ddb66f11eed587beb381249125365a61bd" Oct 30 05:54:24.855974 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.855928 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8e6a5406deb9dd647b72e5013d778ddb66f11eed587beb381249125365a61bd"} err="failed to get container status \"c8e6a5406deb9dd647b72e5013d778ddb66f11eed587beb381249125365a61bd\": rpc error: code = NotFound desc = could not find container \"c8e6a5406deb9dd647b72e5013d778ddb66f11eed587beb381249125365a61bd\": container with ID starting with c8e6a5406deb9dd647b72e5013d778ddb66f11eed587beb381249125365a61bd not found: ID does not exist" Oct 30 05:54:24.855974 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.855941 6999 scope.go:122] "RemoveContainer" containerID="52e85192fd0abb256a2e2c693dcec734355a33580f71e1435f0fe3a128e3ea10" Oct 30 05:54:24.856170 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:24.856155 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52e85192fd0abb256a2e2c693dcec734355a33580f71e1435f0fe3a128e3ea10\": container with ID starting with 52e85192fd0abb256a2e2c693dcec734355a33580f71e1435f0fe3a128e3ea10 not found: ID does not exist" containerID="52e85192fd0abb256a2e2c693dcec734355a33580f71e1435f0fe3a128e3ea10" Oct 30 05:54:24.856207 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.856173 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52e85192fd0abb256a2e2c693dcec734355a33580f71e1435f0fe3a128e3ea10"} err="failed to get container status \"52e85192fd0abb256a2e2c693dcec734355a33580f71e1435f0fe3a128e3ea10\": rpc error: code = NotFound desc = could not find container \"52e85192fd0abb256a2e2c693dcec734355a33580f71e1435f0fe3a128e3ea10\": container with ID starting with 52e85192fd0abb256a2e2c693dcec734355a33580f71e1435f0fe3a128e3ea10 not found: ID does not exist" Oct 30 05:54:24.856207 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.856184 6999 scope.go:122] "RemoveContainer" containerID="107e86592e5a37166c43840d3f66816b76736d29010ed3f147b5b753dffd7c71" Oct 30 05:54:24.856466 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:24.856440 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"107e86592e5a37166c43840d3f66816b76736d29010ed3f147b5b753dffd7c71\": container with ID starting with 107e86592e5a37166c43840d3f66816b76736d29010ed3f147b5b753dffd7c71 not found: ID does not exist" containerID="107e86592e5a37166c43840d3f66816b76736d29010ed3f147b5b753dffd7c71" Oct 30 05:54:24.856550 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.856473 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"107e86592e5a37166c43840d3f66816b76736d29010ed3f147b5b753dffd7c71"} err="failed to get container status \"107e86592e5a37166c43840d3f66816b76736d29010ed3f147b5b753dffd7c71\": rpc error: code = NotFound desc = could not find container \"107e86592e5a37166c43840d3f66816b76736d29010ed3f147b5b753dffd7c71\": container with ID starting with 107e86592e5a37166c43840d3f66816b76736d29010ed3f147b5b753dffd7c71 not found: ID does not exist" Oct 30 05:54:24.856550 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.856490 6999 scope.go:122] "RemoveContainer" containerID="8e344635d04e78d8822773fe6d87d5e8bb56e101731179354bb2b7c7743143ff" Oct 30 05:54:24.856745 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:24.856726 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e344635d04e78d8822773fe6d87d5e8bb56e101731179354bb2b7c7743143ff\": container with ID starting with 8e344635d04e78d8822773fe6d87d5e8bb56e101731179354bb2b7c7743143ff not found: ID does not exist" containerID="8e344635d04e78d8822773fe6d87d5e8bb56e101731179354bb2b7c7743143ff" Oct 30 05:54:24.856811 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.856755 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e344635d04e78d8822773fe6d87d5e8bb56e101731179354bb2b7c7743143ff"} err="failed to get container status \"8e344635d04e78d8822773fe6d87d5e8bb56e101731179354bb2b7c7743143ff\": rpc error: code = NotFound desc = could not find container \"8e344635d04e78d8822773fe6d87d5e8bb56e101731179354bb2b7c7743143ff\": container with ID starting with 8e344635d04e78d8822773fe6d87d5e8bb56e101731179354bb2b7c7743143ff not found: ID does not exist" Oct 30 05:54:24.856811 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.856773 6999 scope.go:122] "RemoveContainer" containerID="dc08fbe805e214511e7b406f3e6f4583f0a548015c82503731efdb53690fa7ab" Oct 30 05:54:24.857060 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:24.857045 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc08fbe805e214511e7b406f3e6f4583f0a548015c82503731efdb53690fa7ab\": container with ID starting with dc08fbe805e214511e7b406f3e6f4583f0a548015c82503731efdb53690fa7ab not found: ID does not exist" containerID="dc08fbe805e214511e7b406f3e6f4583f0a548015c82503731efdb53690fa7ab" Oct 30 05:54:24.857122 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.857065 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc08fbe805e214511e7b406f3e6f4583f0a548015c82503731efdb53690fa7ab"} err="failed to get container status \"dc08fbe805e214511e7b406f3e6f4583f0a548015c82503731efdb53690fa7ab\": rpc error: code = NotFound desc = could not find container \"dc08fbe805e214511e7b406f3e6f4583f0a548015c82503731efdb53690fa7ab\": container with ID starting with dc08fbe805e214511e7b406f3e6f4583f0a548015c82503731efdb53690fa7ab not found: ID does not exist" Oct 30 05:54:24.857122 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.857094 6999 scope.go:122] "RemoveContainer" containerID="1d6fb68ceaffc0f23f79ae294c1490f040e36bda9093c449dd0596e28864d72d" Oct 30 05:54:24.857358 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:24.857342 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d6fb68ceaffc0f23f79ae294c1490f040e36bda9093c449dd0596e28864d72d\": container with ID starting with 1d6fb68ceaffc0f23f79ae294c1490f040e36bda9093c449dd0596e28864d72d not found: ID does not exist" containerID="1d6fb68ceaffc0f23f79ae294c1490f040e36bda9093c449dd0596e28864d72d" Oct 30 05:54:24.857402 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.857363 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d6fb68ceaffc0f23f79ae294c1490f040e36bda9093c449dd0596e28864d72d"} err="failed to get container status \"1d6fb68ceaffc0f23f79ae294c1490f040e36bda9093c449dd0596e28864d72d\": rpc error: code = NotFound desc = could not find container \"1d6fb68ceaffc0f23f79ae294c1490f040e36bda9093c449dd0596e28864d72d\": container with ID starting with 1d6fb68ceaffc0f23f79ae294c1490f040e36bda9093c449dd0596e28864d72d not found: ID does not exist" Oct 30 05:54:24.865883 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.865863 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:54:24.873380 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.873357 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.915455 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.915861 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="alertmanager" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.915887 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="alertmanager" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.915899 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="prom-label-proxy" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.915906 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="prom-label-proxy" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.915918 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="config-reloader" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.915925 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="config-reloader" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.915941 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="kube-rbac-proxy-web" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.915949 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="kube-rbac-proxy-web" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.915965 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="kube-rbac-proxy-metric" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.915971 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="kube-rbac-proxy-metric" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.915995 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="kube-rbac-proxy" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.916001 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="kube-rbac-proxy" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.916013 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="init-config-reloader" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.916020 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="init-config-reloader" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.916032 6999 cpu_manager.go:414] "containerMap: removing container" podUID="19d0e685-73c2-4a16-a54f-b6781627a9c8" containerName="console" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.916039 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19d0e685-73c2-4a16-a54f-b6781627a9c8" containerName="console" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.916104 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="init-config-reloader" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.916115 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="19d0e685-73c2-4a16-a54f-b6781627a9c8" containerName="console" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.916126 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="alertmanager" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.916135 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="prom-label-proxy" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.916142 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="config-reloader" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.916151 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="kube-rbac-proxy-web" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.916158 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="kube-rbac-proxy-metric" Oct 30 05:54:24.931962 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.916168 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" containerName="kube-rbac-proxy" Oct 30 05:54:24.935194 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.933629 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:24.955341 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.955317 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:54:24.967985 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.967953 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 05:54:24.968099 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.968014 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 05:54:24.968099 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.968019 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 05:54:24.968099 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.968058 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2wt58\"" Oct 30 05:54:24.968099 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.968075 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 05:54:24.968099 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.968085 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 05:54:24.968099 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.968089 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 05:54:24.968099 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.968075 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 05:54:24.968099 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.968065 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 05:54:24.968522 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:24.968103 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 05:54:25.033322 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.033285 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/678a3a08-9806-4cb8-923f-8f45c7f84869-config-out\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.033322 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.033319 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-web-config\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.033543 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.033336 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.033543 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.033355 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/678a3a08-9806-4cb8-923f-8f45c7f84869-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.033543 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.033491 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.033684 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.033587 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-config-volume\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.033684 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.033635 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht6mw\" (UniqueName: \"kubernetes.io/projected/678a3a08-9806-4cb8-923f-8f45c7f84869-kube-api-access-ht6mw\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.033684 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.033674 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/678a3a08-9806-4cb8-923f-8f45c7f84869-tls-assets\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.033795 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.033714 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/678a3a08-9806-4cb8-923f-8f45c7f84869-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.033831 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.033791 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.033831 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.033820 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/678a3a08-9806-4cb8-923f-8f45c7f84869-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.033890 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.033853 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.033921 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.033900 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.134427 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.134404 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/678a3a08-9806-4cb8-923f-8f45c7f84869-config-out\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.134540 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.134434 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-web-config\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.134540 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.134450 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.134540 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.134468 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/678a3a08-9806-4cb8-923f-8f45c7f84869-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.134540 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.134494 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.134736 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.134615 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-config-volume\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.134736 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.134640 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht6mw\" (UniqueName: \"kubernetes.io/projected/678a3a08-9806-4cb8-923f-8f45c7f84869-kube-api-access-ht6mw\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.134736 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.134667 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/678a3a08-9806-4cb8-923f-8f45c7f84869-tls-assets\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.134736 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.134704 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/678a3a08-9806-4cb8-923f-8f45c7f84869-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.134923 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.134735 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.134923 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.134764 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/678a3a08-9806-4cb8-923f-8f45c7f84869-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.134923 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.134792 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.134923 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.134798 6999 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 05:54:25.134923 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.134852 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.135495 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.135380 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/678a3a08-9806-4cb8-923f-8f45c7f84869-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.136372 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.136346 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/678a3a08-9806-4cb8-923f-8f45c7f84869-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.136631 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.136606 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/678a3a08-9806-4cb8-923f-8f45c7f84869-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.137741 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.137714 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/678a3a08-9806-4cb8-923f-8f45c7f84869-tls-assets\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.137844 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.137823 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.137885 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.137870 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.137988 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.137968 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.138030 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.137969 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-web-config\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.138180 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.138164 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/678a3a08-9806-4cb8-923f-8f45c7f84869-config-out\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.138480 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.138463 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-config-volume\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.138838 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.138820 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.139830 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.139804 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/678a3a08-9806-4cb8-923f-8f45c7f84869-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.159469 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.159449 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht6mw\" (UniqueName: \"kubernetes.io/projected/678a3a08-9806-4cb8-923f-8f45c7f84869-kube-api-access-ht6mw\") pod \"alertmanager-main-0\" (UID: \"678a3a08-9806-4cb8-923f-8f45c7f84869\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.252005 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.251981 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:54:25.606428 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.606397 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b21885d3-9120-4c95-81ad-cdb80fcdf022" path="/var/lib/kubelet/pods/b21885d3-9120-4c95-81ad-cdb80fcdf022/volumes" Oct 30 05:54:25.622955 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:54:25.622925 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678a3a08_9806_4cb8_923f_8f45c7f84869.slice/crio-e6223a796c8997f142984e40ba0eec44e003e4f70c915e299979ffe853689d45 WatchSource:0}: Error finding container e6223a796c8997f142984e40ba0eec44e003e4f70c915e299979ffe853689d45: Status 404 returned error can't find the container with id e6223a796c8997f142984e40ba0eec44e003e4f70c915e299979ffe853689d45 Oct 30 05:54:25.624922 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.624902 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:54:25.812382 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.812350 6999 generic.go:356] "Generic (PLEG): container finished" podID="678a3a08-9806-4cb8-923f-8f45c7f84869" containerID="0cccf68611bfa8402538404034a9ffc07949c4ab9e5beb272fdf6295eb7eff83" exitCode=0 Oct 30 05:54:25.812519 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.812415 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678a3a08-9806-4cb8-923f-8f45c7f84869","Type":"ContainerDied","Data":"0cccf68611bfa8402538404034a9ffc07949c4ab9e5beb272fdf6295eb7eff83"} Oct 30 05:54:25.812519 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:25.812479 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678a3a08-9806-4cb8-923f-8f45c7f84869","Type":"ContainerStarted","Data":"e6223a796c8997f142984e40ba0eec44e003e4f70c915e299979ffe853689d45"} Oct 30 05:54:26.819624 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:26.819547 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678a3a08-9806-4cb8-923f-8f45c7f84869","Type":"ContainerStarted","Data":"0cf4923d075b68dd1b9ca912e97aad751fdb4ffc7f5ab075555829d8667fd65c"} Oct 30 05:54:26.819971 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:26.819633 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678a3a08-9806-4cb8-923f-8f45c7f84869","Type":"ContainerStarted","Data":"e7d6a9bc3b4ea58d93b39334d410c6fcfc3daa248dc9bbce408cbf18125ac66a"} Oct 30 05:54:26.819971 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:26.819659 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678a3a08-9806-4cb8-923f-8f45c7f84869","Type":"ContainerStarted","Data":"60a5fb57b3bef1437732103d1c4604936519a20f50900e085ec20e7a148a4b41"} Oct 30 05:54:26.819971 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:26.819684 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678a3a08-9806-4cb8-923f-8f45c7f84869","Type":"ContainerStarted","Data":"8f2834b19ec0d4c855b9a2e3eee73e8d843ecde479dd2d5c4595bec20fbe3315"} Oct 30 05:54:26.819971 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:26.819706 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678a3a08-9806-4cb8-923f-8f45c7f84869","Type":"ContainerStarted","Data":"4d68649eac1a8a565defd3b147acd833672b294d91c6821bb5333cf113919880"} Oct 30 05:54:26.819971 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:26.819727 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"678a3a08-9806-4cb8-923f-8f45c7f84869","Type":"ContainerStarted","Data":"c1549de57fafd2cceca4334ebf1bee808afa12d2f11b7a82b2e18d894321defc"} Oct 30 05:54:26.890932 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:26.890889 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.878390713 podStartE2EDuration="2.89087754s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.486827ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:26.88881948 +0000 UTC m=+33.397839367" watchObservedRunningTime="2026-10-30 05:54:26.89087754 +0000 UTC m=+33.399897424" Oct 30 05:54:28.049227 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.049179 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:54:28.049622 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.049458 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="prometheus" containerID="cri-o://efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b" gracePeriod=600 Oct 30 05:54:28.049622 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.049510 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="kube-rbac-proxy-web" containerID="cri-o://2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6" gracePeriod=600 Oct 30 05:54:28.049622 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.049531 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f" gracePeriod=600 Oct 30 05:54:28.049622 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.049487 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="thanos-sidecar" containerID="cri-o://68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1" gracePeriod=600 Oct 30 05:54:28.049622 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.049576 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="config-reloader" containerID="cri-o://0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d" gracePeriod=600 Oct 30 05:54:28.049877 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.049482 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="kube-rbac-proxy" containerID="cri-o://5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f" gracePeriod=600 Oct 30 05:54:28.286445 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.286420 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:28.463032 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463007 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-tls-assets\" (UniqueName: \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-tls-assets\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.463198 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463047 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-metrics-client-ca\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.463198 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463069 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.463198 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463111 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-2\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.463198 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463139 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-tls\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.463198 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463183 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-trusted-ca-bundle\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.463524 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463257 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-kube-rbac-proxy\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.463524 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463287 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-kube-api-access-hng9h\" (UniqueName: \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-kube-api-access-hng9h\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.463524 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463407 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-web-config\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-web-config\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.463668 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463533 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-2" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:28.463668 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463569 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-metrics-client-ca" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:28.463772 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463671 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-trusted-ca-bundle" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:28.463772 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463706 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.463772 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463744 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-0\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.463926 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463772 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-grpc-tls\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.463926 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463807 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-kubelet-serving-ca-bundle\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.463926 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463843 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-1\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.463926 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.463891 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-thanos-prometheus-http-client-file\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.464133 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.464096 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-serving-certs-ca-bundle\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.464195 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.464135 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.464195 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.464169 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-metrics-client-certs\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.464485 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.464236 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config-out\" (UniqueName: \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config-out\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.464485 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.464279 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-db\") pod \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\" (UID: \"45c894e3-ad50-4488-8c5c-5a4bd29ed75a\") " Oct 30 05:54:28.464586 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.464542 6999 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-metrics-client-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.464586 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.464561 6999 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.464586 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.464578 6999 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.465512 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.465477 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-db" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:54:28.466105 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.466080 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-tls" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:28.466390 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.466098 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-tls-assets" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:28.466390 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.466158 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-kube-rbac-proxy" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:28.466390 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.466199 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-kube-api-access-hng9h" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "kube-api-access-hng9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:28.466586 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.466397 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-kube-rbac-proxy-web" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:28.466586 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.466524 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-1" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:28.466698 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.466588 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:28.466698 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.466616 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-kubelet-serving-ca-bundle" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:28.467367 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.467337 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-serving-certs-ca-bundle" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:28.467643 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.467601 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-0" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:28.468328 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.468304 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-thanos-prometheus-http-client-file" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:28.468418 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.468391 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-thanos-sidecar-tls" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:28.469053 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.469036 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-metrics-client-certs" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:28.469357 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.469340 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config-out" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:54:28.469357 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.469341 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-grpc-tls" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:28.478721 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.478703 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-web-config" pod "45c894e3-ad50-4488-8c5c-5a4bd29ed75a" (UID: "45c894e3-ad50-4488-8c5c-5a4bd29ed75a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:28.565108 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565079 6999 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-kube-rbac-proxy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565108 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565102 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hng9h\" (UniqueName: \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-kube-api-access-hng9h\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565108 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565112 6999 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-web-config\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565352 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565120 6999 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565352 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565129 6999 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565352 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565138 6999 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-grpc-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565352 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565148 6999 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565352 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565156 6999 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565352 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565165 6999 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565352 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565173 6999 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565352 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565181 6999 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565352 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565189 6999 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-metrics-client-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565352 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565197 6999 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-config-out\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565352 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565205 6999 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-prometheus-k8s-db\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565352 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565229 6999 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-tls-assets\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565352 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565238 6999 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.565352 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.565246 6999 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/45c894e3-ad50-4488-8c5c-5a4bd29ed75a-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 05:54:28.832867 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.832776 6999 generic.go:356] "Generic (PLEG): container finished" podID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerID="8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f" exitCode=0 Oct 30 05:54:28.832867 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.832807 6999 generic.go:356] "Generic (PLEG): container finished" podID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerID="5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f" exitCode=0 Oct 30 05:54:28.832867 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.832817 6999 generic.go:356] "Generic (PLEG): container finished" podID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerID="2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6" exitCode=0 Oct 30 05:54:28.832867 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.832825 6999 generic.go:356] "Generic (PLEG): container finished" podID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerID="68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1" exitCode=0 Oct 30 05:54:28.832867 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.832833 6999 generic.go:356] "Generic (PLEG): container finished" podID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerID="0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d" exitCode=0 Oct 30 05:54:28.832867 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.832840 6999 generic.go:356] "Generic (PLEG): container finished" podID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerID="efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b" exitCode=0 Oct 30 05:54:28.833286 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.832864 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45c894e3-ad50-4488-8c5c-5a4bd29ed75a","Type":"ContainerDied","Data":"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f"} Oct 30 05:54:28.833286 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.832882 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:28.833286 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.833027 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45c894e3-ad50-4488-8c5c-5a4bd29ed75a","Type":"ContainerDied","Data":"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f"} Oct 30 05:54:28.833286 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.833098 6999 scope.go:122] "RemoveContainer" containerID="8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f" Oct 30 05:54:28.833286 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.833120 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45c894e3-ad50-4488-8c5c-5a4bd29ed75a","Type":"ContainerDied","Data":"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6"} Oct 30 05:54:28.833286 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.833207 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45c894e3-ad50-4488-8c5c-5a4bd29ed75a","Type":"ContainerDied","Data":"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1"} Oct 30 05:54:28.833554 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.833297 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45c894e3-ad50-4488-8c5c-5a4bd29ed75a","Type":"ContainerDied","Data":"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d"} Oct 30 05:54:28.833554 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.833357 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45c894e3-ad50-4488-8c5c-5a4bd29ed75a","Type":"ContainerDied","Data":"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b"} Oct 30 05:54:28.833554 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.833416 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"45c894e3-ad50-4488-8c5c-5a4bd29ed75a","Type":"ContainerDied","Data":"9c33e658395cfc1d8cabf67c663a80f28379c243e36e7d88f4f3a2f0361567fe"} Oct 30 05:54:28.841109 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.841091 6999 scope.go:122] "RemoveContainer" containerID="5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f" Oct 30 05:54:28.847877 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.847860 6999 scope.go:122] "RemoveContainer" containerID="2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6" Oct 30 05:54:28.854342 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.854326 6999 scope.go:122] "RemoveContainer" containerID="68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1" Oct 30 05:54:28.860601 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.860585 6999 scope.go:122] "RemoveContainer" containerID="0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d" Oct 30 05:54:28.869289 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.869273 6999 scope.go:122] "RemoveContainer" containerID="efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b" Oct 30 05:54:28.876458 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.876444 6999 scope.go:122] "RemoveContainer" containerID="41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a" Oct 30 05:54:28.883496 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.883481 6999 scope.go:122] "RemoveContainer" containerID="8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f" Oct 30 05:54:28.883766 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:28.883747 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f\": container with ID starting with 8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f not found: ID does not exist" containerID="8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f" Oct 30 05:54:28.883838 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.883778 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f"} err="failed to get container status \"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f\": rpc error: code = NotFound desc = could not find container \"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f\": container with ID starting with 8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f not found: ID does not exist" Oct 30 05:54:28.883838 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.883803 6999 scope.go:122] "RemoveContainer" containerID="5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f" Oct 30 05:54:28.884077 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:28.884056 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f\": container with ID starting with 5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f not found: ID does not exist" containerID="5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f" Oct 30 05:54:28.884113 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.884087 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f"} err="failed to get container status \"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f\": rpc error: code = NotFound desc = could not find container \"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f\": container with ID starting with 5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f not found: ID does not exist" Oct 30 05:54:28.884113 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.884102 6999 scope.go:122] "RemoveContainer" containerID="2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6" Oct 30 05:54:28.884324 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:28.884308 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6\": container with ID starting with 2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6 not found: ID does not exist" containerID="2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6" Oct 30 05:54:28.884363 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.884327 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6"} err="failed to get container status \"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6\": rpc error: code = NotFound desc = could not find container \"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6\": container with ID starting with 2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6 not found: ID does not exist" Oct 30 05:54:28.884363 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.884338 6999 scope.go:122] "RemoveContainer" containerID="68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1" Oct 30 05:54:28.884581 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:28.884565 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1\": container with ID starting with 68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1 not found: ID does not exist" containerID="68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1" Oct 30 05:54:28.884645 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.884589 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1"} err="failed to get container status \"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1\": rpc error: code = NotFound desc = could not find container \"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1\": container with ID starting with 68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1 not found: ID does not exist" Oct 30 05:54:28.884645 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.884608 6999 scope.go:122] "RemoveContainer" containerID="0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d" Oct 30 05:54:28.884829 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:28.884814 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d\": container with ID starting with 0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d not found: ID does not exist" containerID="0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d" Oct 30 05:54:28.884863 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.884833 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d"} err="failed to get container status \"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d\": rpc error: code = NotFound desc = could not find container \"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d\": container with ID starting with 0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d not found: ID does not exist" Oct 30 05:54:28.884863 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.884845 6999 scope.go:122] "RemoveContainer" containerID="efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b" Oct 30 05:54:28.885033 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:28.885019 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b\": container with ID starting with efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b not found: ID does not exist" containerID="efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b" Oct 30 05:54:28.885077 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.885036 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b"} err="failed to get container status \"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b\": rpc error: code = NotFound desc = could not find container \"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b\": container with ID starting with efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b not found: ID does not exist" Oct 30 05:54:28.885077 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.885048 6999 scope.go:122] "RemoveContainer" containerID="41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a" Oct 30 05:54:28.885259 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:28.885244 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a\": container with ID starting with 41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a not found: ID does not exist" containerID="41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a" Oct 30 05:54:28.885301 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.885262 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a"} err="failed to get container status \"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a\": rpc error: code = NotFound desc = could not find container \"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a\": container with ID starting with 41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a not found: ID does not exist" Oct 30 05:54:28.885301 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.885276 6999 scope.go:122] "RemoveContainer" containerID="8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f" Oct 30 05:54:28.885505 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.885488 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f"} err="failed to get container status \"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f\": rpc error: code = NotFound desc = could not find container \"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f\": container with ID starting with 8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f not found: ID does not exist" Oct 30 05:54:28.885579 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.885506 6999 scope.go:122] "RemoveContainer" containerID="5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f" Oct 30 05:54:28.885738 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.885720 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f"} err="failed to get container status \"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f\": rpc error: code = NotFound desc = could not find container \"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f\": container with ID starting with 5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f not found: ID does not exist" Oct 30 05:54:28.885810 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.885739 6999 scope.go:122] "RemoveContainer" containerID="2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6" Oct 30 05:54:28.885970 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.885952 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6"} err="failed to get container status \"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6\": rpc error: code = NotFound desc = could not find container \"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6\": container with ID starting with 2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6 not found: ID does not exist" Oct 30 05:54:28.886014 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.885971 6999 scope.go:122] "RemoveContainer" containerID="68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1" Oct 30 05:54:28.886193 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.886176 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1"} err="failed to get container status \"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1\": rpc error: code = NotFound desc = could not find container \"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1\": container with ID starting with 68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1 not found: ID does not exist" Oct 30 05:54:28.886282 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.886193 6999 scope.go:122] "RemoveContainer" containerID="0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d" Oct 30 05:54:28.886431 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.886413 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d"} err="failed to get container status \"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d\": rpc error: code = NotFound desc = could not find container \"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d\": container with ID starting with 0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d not found: ID does not exist" Oct 30 05:54:28.886479 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.886432 6999 scope.go:122] "RemoveContainer" containerID="efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b" Oct 30 05:54:28.886661 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.886646 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b"} err="failed to get container status \"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b\": rpc error: code = NotFound desc = could not find container \"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b\": container with ID starting with efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b not found: ID does not exist" Oct 30 05:54:28.886704 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.886661 6999 scope.go:122] "RemoveContainer" containerID="41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a" Oct 30 05:54:28.886890 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.886870 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a"} err="failed to get container status \"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a\": rpc error: code = NotFound desc = could not find container \"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a\": container with ID starting with 41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a not found: ID does not exist" Oct 30 05:54:28.886890 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.886890 6999 scope.go:122] "RemoveContainer" containerID="8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f" Oct 30 05:54:28.887108 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.887093 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f"} err="failed to get container status \"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f\": rpc error: code = NotFound desc = could not find container \"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f\": container with ID starting with 8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f not found: ID does not exist" Oct 30 05:54:28.887108 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.887108 6999 scope.go:122] "RemoveContainer" containerID="5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f" Oct 30 05:54:28.887359 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.887339 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f"} err="failed to get container status \"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f\": rpc error: code = NotFound desc = could not find container \"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f\": container with ID starting with 5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f not found: ID does not exist" Oct 30 05:54:28.887421 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.887360 6999 scope.go:122] "RemoveContainer" containerID="2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6" Oct 30 05:54:28.887571 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.887555 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6"} err="failed to get container status \"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6\": rpc error: code = NotFound desc = could not find container \"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6\": container with ID starting with 2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6 not found: ID does not exist" Oct 30 05:54:28.887571 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.887570 6999 scope.go:122] "RemoveContainer" containerID="68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1" Oct 30 05:54:28.887793 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.887774 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1"} err="failed to get container status \"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1\": rpc error: code = NotFound desc = could not find container \"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1\": container with ID starting with 68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1 not found: ID does not exist" Oct 30 05:54:28.887835 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.887793 6999 scope.go:122] "RemoveContainer" containerID="0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d" Oct 30 05:54:28.888009 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.887992 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d"} err="failed to get container status \"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d\": rpc error: code = NotFound desc = could not find container \"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d\": container with ID starting with 0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d not found: ID does not exist" Oct 30 05:54:28.888050 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.888010 6999 scope.go:122] "RemoveContainer" containerID="efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b" Oct 30 05:54:28.888200 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.888178 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b"} err="failed to get container status \"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b\": rpc error: code = NotFound desc = could not find container \"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b\": container with ID starting with efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b not found: ID does not exist" Oct 30 05:54:28.888297 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.888200 6999 scope.go:122] "RemoveContainer" containerID="41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a" Oct 30 05:54:28.888465 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.888449 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a"} err="failed to get container status \"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a\": rpc error: code = NotFound desc = could not find container \"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a\": container with ID starting with 41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a not found: ID does not exist" Oct 30 05:54:28.888502 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.888465 6999 scope.go:122] "RemoveContainer" containerID="8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f" Oct 30 05:54:28.888695 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.888677 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f"} err="failed to get container status \"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f\": rpc error: code = NotFound desc = could not find container \"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f\": container with ID starting with 8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f not found: ID does not exist" Oct 30 05:54:28.888736 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.888696 6999 scope.go:122] "RemoveContainer" containerID="5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f" Oct 30 05:54:28.888889 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.888874 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f"} err="failed to get container status \"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f\": rpc error: code = NotFound desc = could not find container \"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f\": container with ID starting with 5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f not found: ID does not exist" Oct 30 05:54:28.888925 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.888888 6999 scope.go:122] "RemoveContainer" containerID="2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6" Oct 30 05:54:28.889083 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.889067 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6"} err="failed to get container status \"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6\": rpc error: code = NotFound desc = could not find container \"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6\": container with ID starting with 2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6 not found: ID does not exist" Oct 30 05:54:28.889123 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.889083 6999 scope.go:122] "RemoveContainer" containerID="68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1" Oct 30 05:54:28.889276 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.889260 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1"} err="failed to get container status \"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1\": rpc error: code = NotFound desc = could not find container \"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1\": container with ID starting with 68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1 not found: ID does not exist" Oct 30 05:54:28.889318 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.889276 6999 scope.go:122] "RemoveContainer" containerID="0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d" Oct 30 05:54:28.889490 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.889475 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d"} err="failed to get container status \"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d\": rpc error: code = NotFound desc = could not find container \"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d\": container with ID starting with 0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d not found: ID does not exist" Oct 30 05:54:28.889490 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.889489 6999 scope.go:122] "RemoveContainer" containerID="efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b" Oct 30 05:54:28.889686 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.889670 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b"} err="failed to get container status \"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b\": rpc error: code = NotFound desc = could not find container \"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b\": container with ID starting with efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b not found: ID does not exist" Oct 30 05:54:28.889686 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.889684 6999 scope.go:122] "RemoveContainer" containerID="41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a" Oct 30 05:54:28.889892 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.889877 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a"} err="failed to get container status \"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a\": rpc error: code = NotFound desc = could not find container \"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a\": container with ID starting with 41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a not found: ID does not exist" Oct 30 05:54:28.889892 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.889892 6999 scope.go:122] "RemoveContainer" containerID="8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f" Oct 30 05:54:28.890103 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.890089 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f"} err="failed to get container status \"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f\": rpc error: code = NotFound desc = could not find container \"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f\": container with ID starting with 8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f not found: ID does not exist" Oct 30 05:54:28.890103 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.890103 6999 scope.go:122] "RemoveContainer" containerID="5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f" Oct 30 05:54:28.890333 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.890318 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f"} err="failed to get container status \"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f\": rpc error: code = NotFound desc = could not find container \"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f\": container with ID starting with 5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f not found: ID does not exist" Oct 30 05:54:28.890381 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.890335 6999 scope.go:122] "RemoveContainer" containerID="2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6" Oct 30 05:54:28.890529 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.890513 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6"} err="failed to get container status \"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6\": rpc error: code = NotFound desc = could not find container \"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6\": container with ID starting with 2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6 not found: ID does not exist" Oct 30 05:54:28.890565 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.890528 6999 scope.go:122] "RemoveContainer" containerID="68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1" Oct 30 05:54:28.890708 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.890692 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1"} err="failed to get container status \"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1\": rpc error: code = NotFound desc = could not find container \"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1\": container with ID starting with 68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1 not found: ID does not exist" Oct 30 05:54:28.890748 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.890708 6999 scope.go:122] "RemoveContainer" containerID="0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d" Oct 30 05:54:28.890921 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.890904 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d"} err="failed to get container status \"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d\": rpc error: code = NotFound desc = could not find container \"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d\": container with ID starting with 0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d not found: ID does not exist" Oct 30 05:54:28.890970 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.890922 6999 scope.go:122] "RemoveContainer" containerID="efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b" Oct 30 05:54:28.891110 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.891093 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b"} err="failed to get container status \"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b\": rpc error: code = NotFound desc = could not find container \"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b\": container with ID starting with efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b not found: ID does not exist" Oct 30 05:54:28.891142 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.891111 6999 scope.go:122] "RemoveContainer" containerID="41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a" Oct 30 05:54:28.891322 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.891307 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a"} err="failed to get container status \"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a\": rpc error: code = NotFound desc = could not find container \"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a\": container with ID starting with 41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a not found: ID does not exist" Oct 30 05:54:28.891365 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.891323 6999 scope.go:122] "RemoveContainer" containerID="8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f" Oct 30 05:54:28.891511 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.891497 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f"} err="failed to get container status \"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f\": rpc error: code = NotFound desc = could not find container \"8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f\": container with ID starting with 8655e2ddf78ecff00ccdb05ccfb9e2599219ca9011156e229bc99a15fb3a304f not found: ID does not exist" Oct 30 05:54:28.891559 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.891511 6999 scope.go:122] "RemoveContainer" containerID="5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f" Oct 30 05:54:28.891720 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.891700 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f"} err="failed to get container status \"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f\": rpc error: code = NotFound desc = could not find container \"5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f\": container with ID starting with 5606a3e618f9df91f4b802810d78be13fbfc1b9abe80538d780f36877195380f not found: ID does not exist" Oct 30 05:54:28.891781 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.891721 6999 scope.go:122] "RemoveContainer" containerID="2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6" Oct 30 05:54:28.891941 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.891926 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6"} err="failed to get container status \"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6\": rpc error: code = NotFound desc = could not find container \"2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6\": container with ID starting with 2b73df1d3e8e86eeabfee2e3df7d8e4321ebfb8b5aa9190e904feedb79e108e6 not found: ID does not exist" Oct 30 05:54:28.891982 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.891941 6999 scope.go:122] "RemoveContainer" containerID="68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1" Oct 30 05:54:28.892123 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.892106 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1"} err="failed to get container status \"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1\": rpc error: code = NotFound desc = could not find container \"68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1\": container with ID starting with 68d4627436c1a2c03c04ef4d031300490d24b46309267ce42811cf4b383c13c1 not found: ID does not exist" Oct 30 05:54:28.892190 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.892123 6999 scope.go:122] "RemoveContainer" containerID="0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d" Oct 30 05:54:28.892345 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.892329 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d"} err="failed to get container status \"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d\": rpc error: code = NotFound desc = could not find container \"0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d\": container with ID starting with 0d7a831a9c0728005d6a3e53b52a6f8730161d3c9b98e095b62c7772e89e194d not found: ID does not exist" Oct 30 05:54:28.892394 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.892345 6999 scope.go:122] "RemoveContainer" containerID="efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b" Oct 30 05:54:28.892548 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.892531 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b"} err="failed to get container status \"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b\": rpc error: code = NotFound desc = could not find container \"efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b\": container with ID starting with efc5bf8ce6fefecc89b82792195ec700841460598abcdcda7fd37bb871cdfb9b not found: ID does not exist" Oct 30 05:54:28.892586 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.892547 6999 scope.go:122] "RemoveContainer" containerID="41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a" Oct 30 05:54:28.892742 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.892727 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a"} err="failed to get container status \"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a\": rpc error: code = NotFound desc = could not find container \"41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a\": container with ID starting with 41d2abb3f595bf8c3440e3ca261655fdfa24ee641138482343f4e681a28f4d4a not found: ID does not exist" Oct 30 05:54:28.907325 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.907307 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:54:28.921590 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.921574 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:54:28.953323 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953304 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:54:28.953580 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953569 6999 cpu_manager.go:414] "containerMap: removing container" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="config-reloader" Oct 30 05:54:28.953617 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953581 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="config-reloader" Oct 30 05:54:28.953617 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953595 6999 cpu_manager.go:414] "containerMap: removing container" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="prometheus" Oct 30 05:54:28.953617 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953599 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="prometheus" Oct 30 05:54:28.953617 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953613 6999 cpu_manager.go:414] "containerMap: removing container" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="init-config-reloader" Oct 30 05:54:28.953731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953618 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="init-config-reloader" Oct 30 05:54:28.953731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953625 6999 cpu_manager.go:414] "containerMap: removing container" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="kube-rbac-proxy-thanos" Oct 30 05:54:28.953731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953630 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="kube-rbac-proxy-thanos" Oct 30 05:54:28.953731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953638 6999 cpu_manager.go:414] "containerMap: removing container" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="kube-rbac-proxy" Oct 30 05:54:28.953731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953642 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="kube-rbac-proxy" Oct 30 05:54:28.953731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953649 6999 cpu_manager.go:414] "containerMap: removing container" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="kube-rbac-proxy-web" Oct 30 05:54:28.953731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953653 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="kube-rbac-proxy-web" Oct 30 05:54:28.953731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953663 6999 cpu_manager.go:414] "containerMap: removing container" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="thanos-sidecar" Oct 30 05:54:28.953731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953667 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="thanos-sidecar" Oct 30 05:54:28.953731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953712 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="config-reloader" Oct 30 05:54:28.953731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953720 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="prometheus" Oct 30 05:54:28.953731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953727 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="init-config-reloader" Oct 30 05:54:28.953731 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953733 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="kube-rbac-proxy-thanos" Oct 30 05:54:28.954076 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953739 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="kube-rbac-proxy" Oct 30 05:54:28.954076 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953743 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="kube-rbac-proxy-web" Oct 30 05:54:28.954076 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.953750 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" containerName="thanos-sidecar" Oct 30 05:54:28.957093 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.957079 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:28.975381 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.975358 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 05:54:28.975381 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.975367 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9x2zl\"" Oct 30 05:54:28.976490 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.976471 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 05:54:28.976490 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.976485 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 05:54:28.976614 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.976474 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 05:54:28.976614 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.976487 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 05:54:28.976614 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.976504 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 05:54:28.976614 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.976486 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 05:54:28.976614 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.976530 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 05:54:28.976614 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.976542 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 05:54:28.976614 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.976535 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 05:54:28.976614 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.976536 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 05:54:28.976614 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.976557 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 05:54:28.976614 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.976574 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-aqh338iohmgj7\"" Oct 30 05:54:28.991794 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.991762 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 05:54:28.991929 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.991898 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 05:54:28.993411 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:28.993391 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:54:29.069060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069020 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069478 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069078 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e60660bc-2e77-4a26-bf45-beab6a89adc7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069478 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069099 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069478 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069149 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069478 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069253 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069478 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069275 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069478 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069302 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069478 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069369 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069478 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069411 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069478 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069446 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069478 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069477 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069983 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069535 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pjmp\" (UniqueName: \"kubernetes.io/projected/e60660bc-2e77-4a26-bf45-beab6a89adc7-kube-api-access-9pjmp\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069983 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069597 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069983 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069634 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069983 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069668 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e60660bc-2e77-4a26-bf45-beab6a89adc7-config-out\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069983 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069692 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069983 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069746 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069983 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069765 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-config\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069983 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069778 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e60660bc-2e77-4a26-bf45-beab6a89adc7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.069983 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.069801 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-web-config\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170265 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170242 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e60660bc-2e77-4a26-bf45-beab6a89adc7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170380 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170272 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170380 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170288 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170380 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170312 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170380 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170333 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170380 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170360 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170621 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170480 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170621 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170522 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170621 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170561 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170621 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170589 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170621 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170615 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pjmp\" (UniqueName: \"kubernetes.io/projected/e60660bc-2e77-4a26-bf45-beab6a89adc7-kube-api-access-9pjmp\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170862 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170662 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170862 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170695 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170862 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170725 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e60660bc-2e77-4a26-bf45-beab6a89adc7-config-out\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170862 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170750 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170862 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170794 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170862 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170822 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-config\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.170862 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170844 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e60660bc-2e77-4a26-bf45-beab6a89adc7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.171250 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170881 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-web-config\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.171250 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.170922 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.171250 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.171040 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.171250 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.171148 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.171471 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.171272 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.172680 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.172651 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.173167 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.173142 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.173548 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.173521 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e60660bc-2e77-4a26-bf45-beab6a89adc7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.173763 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.173737 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.173980 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.173954 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.174067 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.173978 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.174067 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.174051 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e60660bc-2e77-4a26-bf45-beab6a89adc7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.175432 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.175008 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.175432 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.175028 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.175432 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.175392 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.175822 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.175801 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e60660bc-2e77-4a26-bf45-beab6a89adc7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.176327 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.176302 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e60660bc-2e77-4a26-bf45-beab6a89adc7-config-out\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.176421 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.176352 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-web-config\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.176421 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.176409 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-config\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.177453 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.177432 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.177559 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.177460 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e60660bc-2e77-4a26-bf45-beab6a89adc7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.197070 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.197052 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pjmp\" (UniqueName: \"kubernetes.io/projected/e60660bc-2e77-4a26-bf45-beab6a89adc7-kube-api-access-9pjmp\") pod \"prometheus-k8s-0\" (UID: \"e60660bc-2e77-4a26-bf45-beab6a89adc7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.273827 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.273811 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:29.415835 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.415811 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4gz57" Oct 30 05:54:29.590167 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.590143 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:54:29.591301 ip-10-0-131-226 kubenswrapper[6999]: W1030 05:54:29.591274 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode60660bc_2e77_4a26_bf45_beab6a89adc7.slice/crio-cbe87a6df5601c6d24ec480f80478c0bc47f4acb217b834e42ce623090cec2ea WatchSource:0}: Error finding container cbe87a6df5601c6d24ec480f80478c0bc47f4acb217b834e42ce623090cec2ea: Status 404 returned error can't find the container with id cbe87a6df5601c6d24ec480f80478c0bc47f4acb217b834e42ce623090cec2ea Oct 30 05:54:29.605986 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.605906 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45c894e3-ad50-4488-8c5c-5a4bd29ed75a" path="/var/lib/kubelet/pods/45c894e3-ad50-4488-8c5c-5a4bd29ed75a/volumes" Oct 30 05:54:29.842422 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.842320 6999 generic.go:356] "Generic (PLEG): container finished" podID="e60660bc-2e77-4a26-bf45-beab6a89adc7" containerID="fb42891a4b17f3034d3b4c4a2cffde9d48b5c4f2c428b3b3e847f3363932c96f" exitCode=0 Oct 30 05:54:29.842555 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.842416 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e60660bc-2e77-4a26-bf45-beab6a89adc7","Type":"ContainerDied","Data":"fb42891a4b17f3034d3b4c4a2cffde9d48b5c4f2c428b3b3e847f3363932c96f"} Oct 30 05:54:29.842618 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:29.842584 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e60660bc-2e77-4a26-bf45-beab6a89adc7","Type":"ContainerStarted","Data":"cbe87a6df5601c6d24ec480f80478c0bc47f4acb217b834e42ce623090cec2ea"} Oct 30 05:54:30.850691 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:30.850658 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e60660bc-2e77-4a26-bf45-beab6a89adc7","Type":"ContainerStarted","Data":"9f121ceba23d4a8d1c394d43233c1eac9b1d5a47c30f229a124733d4bd278ca1"} Oct 30 05:54:30.851115 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:30.850717 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e60660bc-2e77-4a26-bf45-beab6a89adc7","Type":"ContainerStarted","Data":"70632919721dc1fc52a4452b8ff027b950139835e0aa69718416521ed0172692"} Oct 30 05:54:30.851115 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:30.850743 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e60660bc-2e77-4a26-bf45-beab6a89adc7","Type":"ContainerStarted","Data":"12efcc883d2976c558a543ee005c7b77b4a5b32261872b8d2907896203a87166"} Oct 30 05:54:30.851115 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:30.850767 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e60660bc-2e77-4a26-bf45-beab6a89adc7","Type":"ContainerStarted","Data":"bd51a7fa7ba1eb43c5950d6d05648215856075f87c5ff378e7d55fdf4186e9ae"} Oct 30 05:54:30.851115 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:30.850793 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e60660bc-2e77-4a26-bf45-beab6a89adc7","Type":"ContainerStarted","Data":"8a1e3fd0ad220da1a93c944103e7cd9fbc9af5d3e6cb3b21bea15d81363f492d"} Oct 30 05:54:30.851115 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:30.850816 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e60660bc-2e77-4a26-bf45-beab6a89adc7","Type":"ContainerStarted","Data":"ac46b3e36aa05ddc78717e513c6d138b31b3857badef8f8f043961e0df2d8c87"} Oct 30 05:54:30.938403 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:30.938367 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:54:31.117094 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:31.117064 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:54:34.139731 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:54:34.139691 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 05:54:34.274871 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:34.274848 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:34.714492 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:34.714457 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:54:34.864009 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:34.863977 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:54:35.264669 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:35.264640 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:35.280655 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:35.280635 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:35.382940 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:35.382893 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=7.362760628 podStartE2EDuration="7.382879573s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.118945ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:30.930998617 +0000 UTC m=+37.440018505" watchObservedRunningTime="2026-10-30 05:54:35.382879573 +0000 UTC m=+41.891899451" Oct 30 05:54:35.886094 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:35.886064 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:54:43.009613 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:43.009580 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:54:43.231649 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:43.231610 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:54:56.220259 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:54:56.220179 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:55:04.139862 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:55:04.139825 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 05:55:34.139195 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:55:34.139109 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 05:55:56.334357 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:55:56.334326 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:56:01.289043 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:56:01.289006 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:56:04.140628 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:56:04.140586 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 05:56:19.290768 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:56:19.290741 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:56:25.327060 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:56:25.327026 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:56:34.140411 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:56:34.140371 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 05:56:56.461920 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:56:56.461882 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:57:04.140059 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:57:04.139965 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 05:57:17.846345 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:57:17.846308 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:57:18.093390 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:57:18.093351 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:57:30.702138 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:57:30.702102 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:57:30.871559 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:57:30.871478 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:57:31.036581 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:57:31.036495 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:57:34.140339 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:57:34.140306 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 05:57:56.579680 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:57:56.579645 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:58:04.139824 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:58:04.139788 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 05:58:34.140410 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:58:34.140330 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 05:58:56.665560 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:58:56.665524 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 05:59:04.138999 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:59:04.138965 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 05:59:34.139831 ip-10-0-131-226 kubenswrapper[6999]: E1030 05:59:34.139796 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 05:59:56.838147 ip-10-0-131-226 kubenswrapper[6999]: I1030 05:59:56.838107 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:03.311271 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:03.311201 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:03.542548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:03.542474 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:03.793768 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:03.793741 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:04.140088 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:00:04.140054 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:00:05.420675 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:05.420632 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:05.576575 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:05.576546 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:07.064590 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:07.064559 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:22.523704 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:22.523677 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:22.724716 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:22.724688 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:23.972958 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:23.972925 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:24.144814 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:24.144783 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:24.295984 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:24.295913 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:28.266324 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:28.266291 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:34.139074 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:00:34.139038 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:00:40.010424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:40.010366 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:40.399657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:40.399629 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:40.557145 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:40.557120 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:40.900792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:40.900761 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:41.129400 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.129370 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798"] Oct 30 06:00:41.132965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.132942 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.138977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.138949 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.139086 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.138988 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-tmp\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.139134 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.139083 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.139134 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.139107 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.139244 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.139128 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6d27\" (UniqueName: \"kubernetes.io/projected/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-kube-api-access-d6d27\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.139285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.139256 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:41.139327 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.139306 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-manifests\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.176466 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.176412 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798"] Oct 30 06:00:41.185773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.185757 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 06:00:41.186122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.186098 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 06:00:41.186122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.186111 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-n5fhg\"" Oct 30 06:00:41.186122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.186101 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 06:00:41.186305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.186137 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 06:00:41.186305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.186160 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 06:00:41.240432 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.240408 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.240544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.240434 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-tmp\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.240544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.240466 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.240544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.240481 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.240544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.240499 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6d27\" (UniqueName: \"kubernetes.io/projected/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-kube-api-access-d6d27\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.240544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.240541 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-manifests\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.240955 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.240929 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-manifests\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.243039 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.243021 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-webhook-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.243039 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.243031 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-tmp\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.243153 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.243135 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-metrics-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.243371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.243357 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.270349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.270329 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6d27\" (UniqueName: \"kubernetes.io/projected/8d8273b7-a1cf-4467-bfaf-67be2d76ddd6-kube-api-access-d6d27\") pod \"opendatahub-operator-controller-manager-7f5db49b8c-kr798\" (UID: \"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.381004 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.380976 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:41.447985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.447932 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:41.556222 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.556192 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:41.721328 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.721245 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:41.875333 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:00:41.875302 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8273b7_a1cf_4467_bfaf_67be2d76ddd6.slice/crio-89a06ce08efe9ca9d97d129a1b63614db058e9ba243df0e94b5d67ac7f4e41b7 WatchSource:0}: Error finding container 89a06ce08efe9ca9d97d129a1b63614db058e9ba243df0e94b5d67ac7f4e41b7: Status 404 returned error can't find the container with id 89a06ce08efe9ca9d97d129a1b63614db058e9ba243df0e94b5d67ac7f4e41b7 Oct 30 06:00:41.877021 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.877006 6999 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:00:41.885604 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:41.885579 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798"] Oct 30 06:00:42.145752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:42.145724 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" event={"ID":"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6","Type":"ContainerStarted","Data":"89a06ce08efe9ca9d97d129a1b63614db058e9ba243df0e94b5d67ac7f4e41b7"} Oct 30 06:00:43.464137 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:43.464091 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:43.624706 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:43.624670 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:44.307646 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:44.307606 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:44.478540 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:44.478505 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:45.161735 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:45.161707 6999 generic.go:356] "Generic (PLEG): container finished" podID="8d8273b7-a1cf-4467-bfaf-67be2d76ddd6" containerID="c9f02124d43fa094360788f916584560c9ebb2c004b06c427e81be12a568ddc7" exitCode=0 Oct 30 06:00:45.161858 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:45.161749 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" event={"ID":"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6","Type":"ContainerDied","Data":"c9f02124d43fa094360788f916584560c9ebb2c004b06c427e81be12a568ddc7"} Oct 30 06:00:46.168764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:46.168726 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" event={"ID":"8d8273b7-a1cf-4467-bfaf-67be2d76ddd6","Type":"ContainerStarted","Data":"345ed3428baa40172002034599e7293dd4197e40eaf2c487ffc3b3f5c6c64175"} Oct 30 06:00:46.169182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:46.168881 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:46.265720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:46.265672 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" podStartSLOduration=1.7282435170000001 podStartE2EDuration="5.265656917s" totalImagesPullingTime="3.061479078s" totalInitContainerRuntime="475.934322ms" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:46.253483911 +0000 UTC m=+412.762503800" watchObservedRunningTime="2026-10-30 06:00:46.265656917 +0000 UTC m=+412.774676808" Oct 30 06:00:46.933746 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:46.933714 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-59qn9"] Oct 30 06:00:46.937055 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:46.937018 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59qn9" Oct 30 06:00:46.964653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:46.964629 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 06:00:46.974532 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:46.974511 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 06:00:46.974637 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:46.974553 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-vfxtk\"" Oct 30 06:00:46.975245 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:46.975195 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-59qn9"] Oct 30 06:00:46.986440 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:46.986423 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc8h2\" (UniqueName: \"kubernetes.io/projected/9c82678b-8fdf-42f2-b4f3-569d4aae2b61-kube-api-access-qc8h2\") pod \"jobset-operator-674cd59dc5-59qn9\" (UID: \"9c82678b-8fdf-42f2-b4f3-569d4aae2b61\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59qn9" Oct 30 06:00:46.986510 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:46.986472 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c82678b-8fdf-42f2-b4f3-569d4aae2b61-tmp\") pod \"jobset-operator-674cd59dc5-59qn9\" (UID: \"9c82678b-8fdf-42f2-b4f3-569d4aae2b61\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59qn9" Oct 30 06:00:47.087481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:47.087456 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc8h2\" (UniqueName: \"kubernetes.io/projected/9c82678b-8fdf-42f2-b4f3-569d4aae2b61-kube-api-access-qc8h2\") pod \"jobset-operator-674cd59dc5-59qn9\" (UID: \"9c82678b-8fdf-42f2-b4f3-569d4aae2b61\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59qn9" Oct 30 06:00:47.087570 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:47.087508 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c82678b-8fdf-42f2-b4f3-569d4aae2b61-tmp\") pod \"jobset-operator-674cd59dc5-59qn9\" (UID: \"9c82678b-8fdf-42f2-b4f3-569d4aae2b61\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59qn9" Oct 30 06:00:47.087877 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:47.087860 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c82678b-8fdf-42f2-b4f3-569d4aae2b61-tmp\") pod \"jobset-operator-674cd59dc5-59qn9\" (UID: \"9c82678b-8fdf-42f2-b4f3-569d4aae2b61\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59qn9" Oct 30 06:00:47.111070 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:47.111041 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc8h2\" (UniqueName: \"kubernetes.io/projected/9c82678b-8fdf-42f2-b4f3-569d4aae2b61-kube-api-access-qc8h2\") pod \"jobset-operator-674cd59dc5-59qn9\" (UID: \"9c82678b-8fdf-42f2-b4f3-569d4aae2b61\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59qn9" Oct 30 06:00:47.259591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:47.259536 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59qn9" Oct 30 06:00:47.566869 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:47.566841 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-59qn9"] Oct 30 06:00:47.567887 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:00:47.567855 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c82678b_8fdf_42f2_b4f3_569d4aae2b61.slice/crio-f404e77227817fc0a96dd45a56655b6e06b0761913e1996c5ee78c50acf6c74b WatchSource:0}: Error finding container f404e77227817fc0a96dd45a56655b6e06b0761913e1996c5ee78c50acf6c74b: Status 404 returned error can't find the container with id f404e77227817fc0a96dd45a56655b6e06b0761913e1996c5ee78c50acf6c74b Oct 30 06:00:48.177021 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:48.176989 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59qn9" event={"ID":"9c82678b-8fdf-42f2-b4f3-569d4aae2b61","Type":"ContainerStarted","Data":"f404e77227817fc0a96dd45a56655b6e06b0761913e1996c5ee78c50acf6c74b"} Oct 30 06:00:48.419884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:48.419846 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:55.274694 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:55.274654 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:56.423540 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:56.423502 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:56.859425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:56.859388 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:57.067513 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:57.067478 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:57.175120 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:57.175038 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7f5db49b8c-kr798" Oct 30 06:00:57.482605 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:57.482527 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr"] Oct 30 06:00:57.493807 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:57.493782 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr" Oct 30 06:00:57.512188 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:57.512164 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr"] Oct 30 06:00:57.514852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:57.514831 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-z49ns\"" Oct 30 06:00:57.514852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:57.514831 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 06:00:57.515029 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:57.514928 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 06:00:57.578326 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:57.578300 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-984h2\" (UniqueName: \"kubernetes.io/projected/f12cbef7-5c92-4099-854f-db787d010bc4-kube-api-access-984h2\") pod \"cert-manager-webhook-5dbbbf75cc-cfmlr\" (UID: \"f12cbef7-5c92-4099-854f-db787d010bc4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr" Oct 30 06:00:57.578466 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:57.578354 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f12cbef7-5c92-4099-854f-db787d010bc4-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-cfmlr\" (UID: \"f12cbef7-5c92-4099-854f-db787d010bc4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr" Oct 30 06:00:57.679046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:57.679015 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-984h2\" (UniqueName: \"kubernetes.io/projected/f12cbef7-5c92-4099-854f-db787d010bc4-kube-api-access-984h2\") pod \"cert-manager-webhook-5dbbbf75cc-cfmlr\" (UID: \"f12cbef7-5c92-4099-854f-db787d010bc4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr" Oct 30 06:00:57.679233 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:57.679059 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f12cbef7-5c92-4099-854f-db787d010bc4-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-cfmlr\" (UID: \"f12cbef7-5c92-4099-854f-db787d010bc4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr" Oct 30 06:00:57.704016 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:57.703993 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-984h2\" (UniqueName: \"kubernetes.io/projected/f12cbef7-5c92-4099-854f-db787d010bc4-kube-api-access-984h2\") pod \"cert-manager-webhook-5dbbbf75cc-cfmlr\" (UID: \"f12cbef7-5c92-4099-854f-db787d010bc4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr" Oct 30 06:00:57.757171 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:57.757110 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f12cbef7-5c92-4099-854f-db787d010bc4-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-cfmlr\" (UID: \"f12cbef7-5c92-4099-854f-db787d010bc4\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr" Oct 30 06:00:57.806229 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:57.806192 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr" Oct 30 06:00:58.133074 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:58.133044 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr"] Oct 30 06:00:58.137327 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:00:58.137293 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf12cbef7_5c92_4099_854f_db787d010bc4.slice/crio-10c7d44a5d7ecefce25acadee58c2293ae85fc64aa934140bdc8c1487b260d9b WatchSource:0}: Error finding container 10c7d44a5d7ecefce25acadee58c2293ae85fc64aa934140bdc8c1487b260d9b: Status 404 returned error can't find the container with id 10c7d44a5d7ecefce25acadee58c2293ae85fc64aa934140bdc8c1487b260d9b Oct 30 06:00:58.220781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:58.220746 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr" event={"ID":"f12cbef7-5c92-4099-854f-db787d010bc4","Type":"ContainerStarted","Data":"10c7d44a5d7ecefce25acadee58c2293ae85fc64aa934140bdc8c1487b260d9b"} Oct 30 06:00:58.893996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:58.893964 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:59.145597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:59.145485 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:59.323909 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:59.323878 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-m45vq"] Oct 30 06:00:59.330415 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:59.330393 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-m45vq" Oct 30 06:00:59.348650 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:59.348627 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-whtpg\"" Oct 30 06:00:59.361726 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:59.361680 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-m45vq"] Oct 30 06:00:59.368516 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:59.368458 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:00:59.398137 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:59.398081 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/271398cb-63ef-4d6c-9b6f-76141da6c4e7-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-m45vq\" (UID: \"271398cb-63ef-4d6c-9b6f-76141da6c4e7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-m45vq" Oct 30 06:00:59.398279 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:59.398173 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lf95\" (UniqueName: \"kubernetes.io/projected/271398cb-63ef-4d6c-9b6f-76141da6c4e7-kube-api-access-8lf95\") pod \"cert-manager-cainjector-5c57bbd4fc-m45vq\" (UID: \"271398cb-63ef-4d6c-9b6f-76141da6c4e7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-m45vq" Oct 30 06:00:59.499271 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:59.499195 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lf95\" (UniqueName: \"kubernetes.io/projected/271398cb-63ef-4d6c-9b6f-76141da6c4e7-kube-api-access-8lf95\") pod \"cert-manager-cainjector-5c57bbd4fc-m45vq\" (UID: \"271398cb-63ef-4d6c-9b6f-76141da6c4e7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-m45vq" Oct 30 06:00:59.499452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:59.499384 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/271398cb-63ef-4d6c-9b6f-76141da6c4e7-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-m45vq\" (UID: \"271398cb-63ef-4d6c-9b6f-76141da6c4e7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-m45vq" Oct 30 06:00:59.531106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:59.531044 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lf95\" (UniqueName: \"kubernetes.io/projected/271398cb-63ef-4d6c-9b6f-76141da6c4e7-kube-api-access-8lf95\") pod \"cert-manager-cainjector-5c57bbd4fc-m45vq\" (UID: \"271398cb-63ef-4d6c-9b6f-76141da6c4e7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-m45vq" Oct 30 06:00:59.555164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:59.555137 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/271398cb-63ef-4d6c-9b6f-76141da6c4e7-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-m45vq\" (UID: \"271398cb-63ef-4d6c-9b6f-76141da6c4e7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-m45vq" Oct 30 06:00:59.648266 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:59.648173 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-m45vq" Oct 30 06:00:59.973846 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:00:59.973822 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-m45vq"] Oct 30 06:00:59.977418 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:00:59.976498 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod271398cb_63ef_4d6c_9b6f_76141da6c4e7.slice/crio-d1e755991c9fddd570038ecc1c5b553dc8695d7c4d7372cbbc4e549a190bf584 WatchSource:0}: Error finding container d1e755991c9fddd570038ecc1c5b553dc8695d7c4d7372cbbc4e549a190bf584: Status 404 returned error can't find the container with id d1e755991c9fddd570038ecc1c5b553dc8695d7c4d7372cbbc4e549a190bf584 Oct 30 06:01:00.031363 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:00.031328 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:00.233594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:00.233505 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-m45vq" event={"ID":"271398cb-63ef-4d6c-9b6f-76141da6c4e7","Type":"ContainerStarted","Data":"d1e755991c9fddd570038ecc1c5b553dc8695d7c4d7372cbbc4e549a190bf584"} Oct 30 06:01:00.851235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:00.851094 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:01.242334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:01.242296 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59qn9" event={"ID":"9c82678b-8fdf-42f2-b4f3-569d4aae2b61","Type":"ContainerStarted","Data":"6ce26cff33609592e2675fe4dd035f5da0b4927adffe4c9d511891f3a98b1e51"} Oct 30 06:01:01.298073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:01.298025 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-59qn9" podStartSLOduration=2.33705756 podStartE2EDuration="15.298008647s" totalImagesPullingTime="12.960951087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:01.295338608 +0000 UTC m=+427.804358494" watchObservedRunningTime="2026-10-30 06:01:01.298008647 +0000 UTC m=+427.807028531" Oct 30 06:01:02.541662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:02.541578 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:03.252182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:03.252143 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr" event={"ID":"f12cbef7-5c92-4099-854f-db787d010bc4","Type":"ContainerStarted","Data":"15af2f378ab1cff00b26da198765a797ec1f321590a18f57b970414dd27e9db8"} Oct 30 06:01:03.252387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:03.252208 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr" Oct 30 06:01:03.253772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:03.253749 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-m45vq" event={"ID":"271398cb-63ef-4d6c-9b6f-76141da6c4e7","Type":"ContainerStarted","Data":"47988ca49d397a60523dcbed5e79598d194035cff6e7871f79a37004d05e1ee1"} Oct 30 06:01:03.299582 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:03.299533 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr" podStartSLOduration=2.1989430739999998 podStartE2EDuration="6.299519518s" totalImagesPullingTime="4.100576444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:03.297276806 +0000 UTC m=+429.806296692" watchObservedRunningTime="2026-10-30 06:01:03.299519518 +0000 UTC m=+429.808539405" Oct 30 06:01:04.142786 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:01:04.142753 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:01:04.333989 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:04.333948 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:04.583726 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:04.583638 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:04.887601 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:04.887565 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:05.128363 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:05.128326 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:06.947482 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:06.947435 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-m45vq" podStartSLOduration=4.875680516 podStartE2EDuration="7.947421964s" totalImagesPullingTime="3.071741448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:03.352749302 +0000 UTC m=+429.861769188" watchObservedRunningTime="2026-10-30 06:01:06.947421964 +0000 UTC m=+433.456441850" Oct 30 06:01:06.948085 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:06.948071 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-twlhp"] Oct 30 06:01:06.951295 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:06.951280 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-twlhp" Oct 30 06:01:06.986170 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:06.986144 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-twlhp"] Oct 30 06:01:06.988299 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:06.988276 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:06.990200 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:06.990182 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qb5zz\"" Oct 30 06:01:07.078556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:07.078531 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmf4z\" (UniqueName: \"kubernetes.io/projected/ca1e9050-b49b-47e7-a2bb-8e193ae11326-kube-api-access-dmf4z\") pod \"cert-manager-597b95cb84-twlhp\" (UID: \"ca1e9050-b49b-47e7-a2bb-8e193ae11326\") " pod="cert-manager/cert-manager-597b95cb84-twlhp" Oct 30 06:01:07.078690 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:07.078593 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca1e9050-b49b-47e7-a2bb-8e193ae11326-bound-sa-token\") pod \"cert-manager-597b95cb84-twlhp\" (UID: \"ca1e9050-b49b-47e7-a2bb-8e193ae11326\") " pod="cert-manager/cert-manager-597b95cb84-twlhp" Oct 30 06:01:07.180123 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:07.180095 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca1e9050-b49b-47e7-a2bb-8e193ae11326-bound-sa-token\") pod \"cert-manager-597b95cb84-twlhp\" (UID: \"ca1e9050-b49b-47e7-a2bb-8e193ae11326\") " pod="cert-manager/cert-manager-597b95cb84-twlhp" Oct 30 06:01:07.180297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:07.180166 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmf4z\" (UniqueName: \"kubernetes.io/projected/ca1e9050-b49b-47e7-a2bb-8e193ae11326-kube-api-access-dmf4z\") pod \"cert-manager-597b95cb84-twlhp\" (UID: \"ca1e9050-b49b-47e7-a2bb-8e193ae11326\") " pod="cert-manager/cert-manager-597b95cb84-twlhp" Oct 30 06:01:07.226953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:07.226886 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ca1e9050-b49b-47e7-a2bb-8e193ae11326-bound-sa-token\") pod \"cert-manager-597b95cb84-twlhp\" (UID: \"ca1e9050-b49b-47e7-a2bb-8e193ae11326\") " pod="cert-manager/cert-manager-597b95cb84-twlhp" Oct 30 06:01:07.227170 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:07.227150 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmf4z\" (UniqueName: \"kubernetes.io/projected/ca1e9050-b49b-47e7-a2bb-8e193ae11326-kube-api-access-dmf4z\") pod \"cert-manager-597b95cb84-twlhp\" (UID: \"ca1e9050-b49b-47e7-a2bb-8e193ae11326\") " pod="cert-manager/cert-manager-597b95cb84-twlhp" Oct 30 06:01:07.262920 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:07.262895 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-twlhp" Oct 30 06:01:07.293571 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:07.293546 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:07.581194 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:07.581168 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-twlhp"] Oct 30 06:01:07.582998 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:01:07.582973 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca1e9050_b49b_47e7_a2bb_8e193ae11326.slice/crio-8cfd34ceb377eecb21e508187a5bf1617ea7720e19402be72eb715989886d87b WatchSource:0}: Error finding container 8cfd34ceb377eecb21e508187a5bf1617ea7720e19402be72eb715989886d87b: Status 404 returned error can't find the container with id 8cfd34ceb377eecb21e508187a5bf1617ea7720e19402be72eb715989886d87b Oct 30 06:01:08.008728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:08.008696 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:08.275912 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:08.275834 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-twlhp" event={"ID":"ca1e9050-b49b-47e7-a2bb-8e193ae11326","Type":"ContainerStarted","Data":"a0447eaf003238209ea4173e4082c76cd5837715a1d707ff06cd97508fc0ba7a"} Oct 30 06:01:08.275912 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:08.275872 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-twlhp" event={"ID":"ca1e9050-b49b-47e7-a2bb-8e193ae11326","Type":"ContainerStarted","Data":"8cfd34ceb377eecb21e508187a5bf1617ea7720e19402be72eb715989886d87b"} Oct 30 06:01:08.332205 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:08.332161 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-twlhp" podStartSLOduration=2.332148234 podStartE2EDuration="2.332148234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:08.330871979 +0000 UTC m=+434.839891866" watchObservedRunningTime="2026-10-30 06:01:08.332148234 +0000 UTC m=+434.841168118" Oct 30 06:01:09.149568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:09.149537 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:09.261092 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:09.261060 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cfmlr" Oct 30 06:01:09.621479 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:09.621450 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:11.330257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:11.330203 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:11.505633 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:11.505599 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:12.110020 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:12.109984 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:12.312630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:12.312601 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:12.602002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:12.601913 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:12.925555 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:12.925519 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847"] Oct 30 06:01:12.929507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:12.929484 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847" Oct 30 06:01:12.931952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:12.931930 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:12.956955 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:12.956933 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 06:01:12.965466 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:12.965441 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 06:01:12.970405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:12.970380 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847"] Oct 30 06:01:13.015793 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.015767 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 06:01:13.016012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.015993 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nr94k\"" Oct 30 06:01:13.016084 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.016036 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 06:01:13.033469 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.033439 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxbr5\" (UniqueName: \"kubernetes.io/projected/0d476729-9d6f-4624-b080-e19e28bcd0f5-kube-api-access-qxbr5\") pod \"kube-auth-proxy-5c4bb65c68-z6847\" (UID: \"0d476729-9d6f-4624-b080-e19e28bcd0f5\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847" Oct 30 06:01:13.033610 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.033471 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d476729-9d6f-4624-b080-e19e28bcd0f5-tmp\") pod \"kube-auth-proxy-5c4bb65c68-z6847\" (UID: \"0d476729-9d6f-4624-b080-e19e28bcd0f5\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847" Oct 30 06:01:13.033668 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.033619 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d476729-9d6f-4624-b080-e19e28bcd0f5-tls-certs\") pod \"kube-auth-proxy-5c4bb65c68-z6847\" (UID: \"0d476729-9d6f-4624-b080-e19e28bcd0f5\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847" Oct 30 06:01:13.134201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.134166 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxbr5\" (UniqueName: \"kubernetes.io/projected/0d476729-9d6f-4624-b080-e19e28bcd0f5-kube-api-access-qxbr5\") pod \"kube-auth-proxy-5c4bb65c68-z6847\" (UID: \"0d476729-9d6f-4624-b080-e19e28bcd0f5\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847" Oct 30 06:01:13.134364 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.134202 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d476729-9d6f-4624-b080-e19e28bcd0f5-tmp\") pod \"kube-auth-proxy-5c4bb65c68-z6847\" (UID: \"0d476729-9d6f-4624-b080-e19e28bcd0f5\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847" Oct 30 06:01:13.134364 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.134285 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d476729-9d6f-4624-b080-e19e28bcd0f5-tls-certs\") pod \"kube-auth-proxy-5c4bb65c68-z6847\" (UID: \"0d476729-9d6f-4624-b080-e19e28bcd0f5\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847" Oct 30 06:01:13.136564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.136535 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0d476729-9d6f-4624-b080-e19e28bcd0f5-tmp\") pod \"kube-auth-proxy-5c4bb65c68-z6847\" (UID: \"0d476729-9d6f-4624-b080-e19e28bcd0f5\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847" Oct 30 06:01:13.136754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.136736 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d476729-9d6f-4624-b080-e19e28bcd0f5-tls-certs\") pod \"kube-auth-proxy-5c4bb65c68-z6847\" (UID: \"0d476729-9d6f-4624-b080-e19e28bcd0f5\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847" Oct 30 06:01:13.166131 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.166100 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxbr5\" (UniqueName: \"kubernetes.io/projected/0d476729-9d6f-4624-b080-e19e28bcd0f5-kube-api-access-qxbr5\") pod \"kube-auth-proxy-5c4bb65c68-z6847\" (UID: \"0d476729-9d6f-4624-b080-e19e28bcd0f5\") " pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847" Oct 30 06:01:13.180764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.180700 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:13.245348 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.245324 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847" Oct 30 06:01:13.393003 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.392902 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:13.570127 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.570104 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847"] Oct 30 06:01:13.571460 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:01:13.571424 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d476729_9d6f_4624_b080_e19e28bcd0f5.slice/crio-f8fa8dce7efaa10bda99a4f24bcfaf20ae05297f4b9735fffdd79775a0e5d83f WatchSource:0}: Error finding container f8fa8dce7efaa10bda99a4f24bcfaf20ae05297f4b9735fffdd79775a0e5d83f: Status 404 returned error can't find the container with id f8fa8dce7efaa10bda99a4f24bcfaf20ae05297f4b9735fffdd79775a0e5d83f Oct 30 06:01:13.581770 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:13.581740 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:14.308190 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:14.308142 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847" event={"ID":"0d476729-9d6f-4624-b080-e19e28bcd0f5","Type":"ContainerStarted","Data":"f8fa8dce7efaa10bda99a4f24bcfaf20ae05297f4b9735fffdd79775a0e5d83f"} Oct 30 06:01:14.802179 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:14.802136 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:15.498609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:15.498572 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:15.686256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:15.683961 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:16.176240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:16.175721 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:16.320538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:16.320461 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847" event={"ID":"0d476729-9d6f-4624-b080-e19e28bcd0f5","Type":"ContainerStarted","Data":"7481841fa4ecb8b8ff666cd2dbac0046b750520f48f8a1151fc37051626ee29c"} Oct 30 06:01:16.368900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:16.368855 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c4bb65c68-z6847" podStartSLOduration=1.917399514 podStartE2EDuration="4.368842411s" totalImagesPullingTime="2.451442897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:16.3673931 +0000 UTC m=+442.876413003" watchObservedRunningTime="2026-10-30 06:01:16.368842411 +0000 UTC m=+442.877862297" Oct 30 06:01:19.409839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.409803 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jl6p5"] Oct 30 06:01:19.414905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.414580 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" Oct 30 06:01:19.443407 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.443385 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jh7qr\"" Oct 30 06:01:19.447042 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.447016 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:19.455977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.455955 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jl6p5"] Oct 30 06:01:19.502329 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.502309 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 06:01:19.502329 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.502314 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 06:01:19.502644 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.502631 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 06:01:19.528176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.528152 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jl6p5"] Oct 30 06:01:19.595607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.595576 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlrlc\" (UniqueName: \"kubernetes.io/projected/3a3bea34-b2eb-4d7f-80a5-4029aa31b98f-kube-api-access-wlrlc\") pod \"limitador-limitador-74dfcfb894-jl6p5\" (UID: \"3a3bea34-b2eb-4d7f-80a5-4029aa31b98f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" Oct 30 06:01:19.595721 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.595624 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a3bea34-b2eb-4d7f-80a5-4029aa31b98f-config-file\") pod \"limitador-limitador-74dfcfb894-jl6p5\" (UID: \"3a3bea34-b2eb-4d7f-80a5-4029aa31b98f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" Oct 30 06:01:19.645566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.645540 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:19.697182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.697110 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrlc\" (UniqueName: \"kubernetes.io/projected/3a3bea34-b2eb-4d7f-80a5-4029aa31b98f-kube-api-access-wlrlc\") pod \"limitador-limitador-74dfcfb894-jl6p5\" (UID: \"3a3bea34-b2eb-4d7f-80a5-4029aa31b98f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" Oct 30 06:01:19.697182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.697157 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a3bea34-b2eb-4d7f-80a5-4029aa31b98f-config-file\") pod \"limitador-limitador-74dfcfb894-jl6p5\" (UID: \"3a3bea34-b2eb-4d7f-80a5-4029aa31b98f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" Oct 30 06:01:19.697838 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.697819 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a3bea34-b2eb-4d7f-80a5-4029aa31b98f-config-file\") pod \"limitador-limitador-74dfcfb894-jl6p5\" (UID: \"3a3bea34-b2eb-4d7f-80a5-4029aa31b98f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" Oct 30 06:01:19.723171 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.723143 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlrlc\" (UniqueName: \"kubernetes.io/projected/3a3bea34-b2eb-4d7f-80a5-4029aa31b98f-kube-api-access-wlrlc\") pod \"limitador-limitador-74dfcfb894-jl6p5\" (UID: \"3a3bea34-b2eb-4d7f-80a5-4029aa31b98f\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" Oct 30 06:01:19.729821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.729801 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" Oct 30 06:01:19.807083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.807053 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:19.970762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:19.970683 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:20.031419 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:20.031393 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jl6p5"] Oct 30 06:01:20.033009 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:01:20.032976 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3bea34_b2eb_4d7f_80a5_4029aa31b98f.slice/crio-7ccd8527471c3517eaa218ed502737edae0b80456aabbaf90a8595b5dc9acd91 WatchSource:0}: Error finding container 7ccd8527471c3517eaa218ed502737edae0b80456aabbaf90a8595b5dc9acd91: Status 404 returned error can't find the container with id 7ccd8527471c3517eaa218ed502737edae0b80456aabbaf90a8595b5dc9acd91 Oct 30 06:01:20.339398 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:20.339317 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" event={"ID":"3a3bea34-b2eb-4d7f-80a5-4029aa31b98f","Type":"ContainerStarted","Data":"7ccd8527471c3517eaa218ed502737edae0b80456aabbaf90a8595b5dc9acd91"} Oct 30 06:01:20.440321 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:20.440275 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:20.665446 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:20.665408 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:21.026955 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:21.026874 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:21.299306 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:21.299233 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:21.557518 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:21.557430 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bqddx"] Oct 30 06:01:21.564097 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:21.564072 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bqddx" Oct 30 06:01:21.581998 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:21.581961 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5khf9\"" Oct 30 06:01:21.595067 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:21.595014 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bqddx"] Oct 30 06:01:21.717343 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:21.717313 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg62x\" (UniqueName: \"kubernetes.io/projected/f8f10885-315c-4312-a86f-06d595294e63-kube-api-access-lg62x\") pod \"authorino-56c6c9ddd5-bqddx\" (UID: \"f8f10885-315c-4312-a86f-06d595294e63\") " pod="kuadrant-system/authorino-56c6c9ddd5-bqddx" Oct 30 06:01:21.818481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:21.818407 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg62x\" (UniqueName: \"kubernetes.io/projected/f8f10885-315c-4312-a86f-06d595294e63-kube-api-access-lg62x\") pod \"authorino-56c6c9ddd5-bqddx\" (UID: \"f8f10885-315c-4312-a86f-06d595294e63\") " pod="kuadrant-system/authorino-56c6c9ddd5-bqddx" Oct 30 06:01:21.848485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:21.848454 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg62x\" (UniqueName: \"kubernetes.io/projected/f8f10885-315c-4312-a86f-06d595294e63-kube-api-access-lg62x\") pod \"authorino-56c6c9ddd5-bqddx\" (UID: \"f8f10885-315c-4312-a86f-06d595294e63\") " pod="kuadrant-system/authorino-56c6c9ddd5-bqddx" Oct 30 06:01:21.878167 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:21.877864 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bqddx" Oct 30 06:01:22.206339 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:22.206315 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bqddx"] Oct 30 06:01:22.208656 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:01:22.208632 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8f10885_315c_4312_a86f_06d595294e63.slice/crio-4b38e180738cf5ca0f644eef01ee83ae905a73f0690becd8631c2d5faa6fba0b WatchSource:0}: Error finding container 4b38e180738cf5ca0f644eef01ee83ae905a73f0690becd8631c2d5faa6fba0b: Status 404 returned error can't find the container with id 4b38e180738cf5ca0f644eef01ee83ae905a73f0690becd8631c2d5faa6fba0b Oct 30 06:01:22.353291 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:22.353257 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bqddx" event={"ID":"f8f10885-315c-4312-a86f-06d595294e63","Type":"ContainerStarted","Data":"4b38e180738cf5ca0f644eef01ee83ae905a73f0690becd8631c2d5faa6fba0b"} Oct 30 06:01:22.354781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:22.354754 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" event={"ID":"3a3bea34-b2eb-4d7f-80a5-4029aa31b98f","Type":"ContainerStarted","Data":"794276e47ed4ea6e39151a422a460cf26d8d34d9d2a77b9c7a8d5aec3a732114"} Oct 30 06:01:22.354908 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:22.354888 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" Oct 30 06:01:22.409998 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:22.409959 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" podStartSLOduration=1.276203511 podStartE2EDuration="3.409947788s" totalImagesPullingTime="2.133744277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:22.409617115 +0000 UTC m=+448.918637007" watchObservedRunningTime="2026-10-30 06:01:22.409947788 +0000 UTC m=+448.918967729" Oct 30 06:01:24.366927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:24.366891 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bqddx" event={"ID":"f8f10885-315c-4312-a86f-06d595294e63","Type":"ContainerStarted","Data":"87b4b53a07c2d24ed7aaa44f63abe62fc0e745c3052dab0f4ea2ca2984dc60c4"} Oct 30 06:01:24.415231 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:24.415155 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-bqddx" podStartSLOduration=1.8876828209999998 podStartE2EDuration="3.415143038s" totalImagesPullingTime="1.527460217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:24.4141723 +0000 UTC m=+450.923192188" watchObservedRunningTime="2026-10-30 06:01:24.415143038 +0000 UTC m=+450.924162926" Oct 30 06:01:28.897863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:28.897831 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:29.195237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:29.195111 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:29.399994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:29.399960 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:31.303441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:31.303405 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:31.457949 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:31.457915 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:33.361587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:33.361553 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" Oct 30 06:01:33.443843 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:33.443770 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jl6p5"] Oct 30 06:01:33.443993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:33.443949 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" podUID="3a3bea34-b2eb-4d7f-80a5-4029aa31b98f" containerName="limitador" containerID="cri-o://794276e47ed4ea6e39151a422a460cf26d8d34d9d2a77b9c7a8d5aec3a732114" gracePeriod=30 Oct 30 06:01:34.017540 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.017485 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" Oct 30 06:01:34.121316 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.121287 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3bea34-b2eb-4d7f-80a5-4029aa31b98f-kube-api-access-wlrlc\" (UniqueName: \"kubernetes.io/projected/3a3bea34-b2eb-4d7f-80a5-4029aa31b98f-kube-api-access-wlrlc\") pod \"3a3bea34-b2eb-4d7f-80a5-4029aa31b98f\" (UID: \"3a3bea34-b2eb-4d7f-80a5-4029aa31b98f\") " Oct 30 06:01:34.121454 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.121356 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a3bea34-b2eb-4d7f-80a5-4029aa31b98f-config-file\" (UniqueName: \"kubernetes.io/configmap/3a3bea34-b2eb-4d7f-80a5-4029aa31b98f-config-file\") pod \"3a3bea34-b2eb-4d7f-80a5-4029aa31b98f\" (UID: \"3a3bea34-b2eb-4d7f-80a5-4029aa31b98f\") " Oct 30 06:01:34.121753 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.121719 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3bea34-b2eb-4d7f-80a5-4029aa31b98f-config-file" pod "3a3bea34-b2eb-4d7f-80a5-4029aa31b98f" (UID: "3a3bea34-b2eb-4d7f-80a5-4029aa31b98f"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:01:34.123580 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.123554 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3bea34-b2eb-4d7f-80a5-4029aa31b98f-kube-api-access-wlrlc" pod "3a3bea34-b2eb-4d7f-80a5-4029aa31b98f" (UID: "3a3bea34-b2eb-4d7f-80a5-4029aa31b98f"). InnerVolumeSpecName "kube-api-access-wlrlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:01:34.140007 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:01:34.139979 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:01:34.222230 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.222190 6999 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3a3bea34-b2eb-4d7f-80a5-4029aa31b98f-config-file\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:01:34.222230 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.222231 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlrlc\" (UniqueName: \"kubernetes.io/projected/3a3bea34-b2eb-4d7f-80a5-4029aa31b98f-kube-api-access-wlrlc\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:01:34.416528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.416499 6999 generic.go:356] "Generic (PLEG): container finished" podID="3a3bea34-b2eb-4d7f-80a5-4029aa31b98f" containerID="794276e47ed4ea6e39151a422a460cf26d8d34d9d2a77b9c7a8d5aec3a732114" exitCode=0 Oct 30 06:01:34.416915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.416549 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" Oct 30 06:01:34.416915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.416588 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" event={"ID":"3a3bea34-b2eb-4d7f-80a5-4029aa31b98f","Type":"ContainerDied","Data":"794276e47ed4ea6e39151a422a460cf26d8d34d9d2a77b9c7a8d5aec3a732114"} Oct 30 06:01:34.416915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.416643 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jl6p5" event={"ID":"3a3bea34-b2eb-4d7f-80a5-4029aa31b98f","Type":"ContainerDied","Data":"7ccd8527471c3517eaa218ed502737edae0b80456aabbaf90a8595b5dc9acd91"} Oct 30 06:01:34.416915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.416670 6999 scope.go:122] "RemoveContainer" containerID="794276e47ed4ea6e39151a422a460cf26d8d34d9d2a77b9c7a8d5aec3a732114" Oct 30 06:01:34.425252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.425196 6999 scope.go:122] "RemoveContainer" containerID="794276e47ed4ea6e39151a422a460cf26d8d34d9d2a77b9c7a8d5aec3a732114" Oct 30 06:01:34.425482 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:01:34.425464 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794276e47ed4ea6e39151a422a460cf26d8d34d9d2a77b9c7a8d5aec3a732114\": container with ID starting with 794276e47ed4ea6e39151a422a460cf26d8d34d9d2a77b9c7a8d5aec3a732114 not found: ID does not exist" containerID="794276e47ed4ea6e39151a422a460cf26d8d34d9d2a77b9c7a8d5aec3a732114" Oct 30 06:01:34.425533 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.425492 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794276e47ed4ea6e39151a422a460cf26d8d34d9d2a77b9c7a8d5aec3a732114"} err="failed to get container status \"794276e47ed4ea6e39151a422a460cf26d8d34d9d2a77b9c7a8d5aec3a732114\": rpc error: code = NotFound desc = could not find container \"794276e47ed4ea6e39151a422a460cf26d8d34d9d2a77b9c7a8d5aec3a732114\": container with ID starting with 794276e47ed4ea6e39151a422a460cf26d8d34d9d2a77b9c7a8d5aec3a732114 not found: ID does not exist" Oct 30 06:01:34.467448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.467427 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jl6p5"] Oct 30 06:01:34.479789 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:34.479767 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jl6p5"] Oct 30 06:01:35.606083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:35.606049 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a3bea34-b2eb-4d7f-80a5-4029aa31b98f" path="/var/lib/kubelet/pods/3a3bea34-b2eb-4d7f-80a5-4029aa31b98f/volumes" Oct 30 06:01:37.840099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:37.840067 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:37.997160 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:37.997105 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:40.315515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:40.315476 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:40.506278 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:40.506244 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:41.170052 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.170016 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82"] Oct 30 06:01:41.170458 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.170441 6999 cpu_manager.go:414] "containerMap: removing container" podUID="3a3bea34-b2eb-4d7f-80a5-4029aa31b98f" containerName="limitador" Oct 30 06:01:41.170557 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.170460 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a3bea34-b2eb-4d7f-80a5-4029aa31b98f" containerName="limitador" Oct 30 06:01:41.170614 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.170600 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a3bea34-b2eb-4d7f-80a5-4029aa31b98f" containerName="limitador" Oct 30 06:01:41.172797 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.172777 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82" Oct 30 06:01:41.212856 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.212837 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-smz27\"" Oct 30 06:01:41.212856 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.212837 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:01:41.224712 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.224694 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 06:01:41.224845 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.224722 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:01:41.225885 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.225863 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82"] Oct 30 06:01:41.277988 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.277964 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z5fx\" (UniqueName: \"kubernetes.io/projected/f3f6f187-d56f-487a-a923-7d6d310a03d1-kube-api-access-9z5fx\") pod \"postgres-7cdbf6dc44-skg82\" (UID: \"f3f6f187-d56f-487a-a923-7d6d310a03d1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82" Oct 30 06:01:41.278073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.278007 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f3f6f187-d56f-487a-a923-7d6d310a03d1-data\") pod \"postgres-7cdbf6dc44-skg82\" (UID: \"f3f6f187-d56f-487a-a923-7d6d310a03d1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82" Oct 30 06:01:41.373578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.373549 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:41.378723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.378703 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z5fx\" (UniqueName: \"kubernetes.io/projected/f3f6f187-d56f-487a-a923-7d6d310a03d1-kube-api-access-9z5fx\") pod \"postgres-7cdbf6dc44-skg82\" (UID: \"f3f6f187-d56f-487a-a923-7d6d310a03d1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82" Oct 30 06:01:41.378782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.378751 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f3f6f187-d56f-487a-a923-7d6d310a03d1-data\") pod \"postgres-7cdbf6dc44-skg82\" (UID: \"f3f6f187-d56f-487a-a923-7d6d310a03d1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82" Oct 30 06:01:41.379132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.379117 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f3f6f187-d56f-487a-a923-7d6d310a03d1-data\") pod \"postgres-7cdbf6dc44-skg82\" (UID: \"f3f6f187-d56f-487a-a923-7d6d310a03d1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82" Oct 30 06:01:41.407184 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.407160 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z5fx\" (UniqueName: \"kubernetes.io/projected/f3f6f187-d56f-487a-a923-7d6d310a03d1-kube-api-access-9z5fx\") pod \"postgres-7cdbf6dc44-skg82\" (UID: \"f3f6f187-d56f-487a-a923-7d6d310a03d1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82" Oct 30 06:01:41.484081 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.484027 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82" Oct 30 06:01:41.782798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:41.782776 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82"] Oct 30 06:01:41.784256 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:01:41.784231 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f6f187_d56f_487a_a923_7d6d310a03d1.slice/crio-019698edc66174504f57837588110e271dd0e83d7ff778b51cf2f5ca0ebcfaf3 WatchSource:0}: Error finding container 019698edc66174504f57837588110e271dd0e83d7ff778b51cf2f5ca0ebcfaf3: Status 404 returned error can't find the container with id 019698edc66174504f57837588110e271dd0e83d7ff778b51cf2f5ca0ebcfaf3 Oct 30 06:01:42.457106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:42.456979 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82" event={"ID":"f3f6f187-d56f-487a-a923-7d6d310a03d1","Type":"ContainerStarted","Data":"019698edc66174504f57837588110e271dd0e83d7ff778b51cf2f5ca0ebcfaf3"} Oct 30 06:01:42.680717 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:42.680663 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:42.853678 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:42.853592 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:43.423651 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:43.423615 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:43.643924 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:43.643893 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:43.859508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:43.859435 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:44.064378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:44.064301 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:47.482804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:47.482764 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82" event={"ID":"f3f6f187-d56f-487a-a923-7d6d310a03d1","Type":"ContainerStarted","Data":"15be51ba2abdc8c9e5922ef1bd3a6839209c7f14b395981be723c2d8371b2df4"} Oct 30 06:01:47.483225 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:47.482926 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82" Oct 30 06:01:52.646371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:52.646332 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:52.819597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:52.819564 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:53.274565 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:53.274539 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:53.474509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:53.474474 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:53.519845 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:53.519825 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82" Oct 30 06:01:53.563646 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:53.563546 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-skg82" podStartSLOduration=7.768401289 podStartE2EDuration="12.563535539s" totalImagesPullingTime="4.79513425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:47.529369069 +0000 UTC m=+474.038388980" watchObservedRunningTime="2026-10-30 06:01:53.563535539 +0000 UTC m=+480.072555448" Oct 30 06:01:56.648899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:56.648865 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:56.787798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:56.787755 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:01:57.127472 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:01:57.127447 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:01.171279 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.171237 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:01.321829 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.321795 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:01.543290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.543184 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:01.720547 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.720508 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:01.777455 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.777425 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5749bffc96-tr2hn"] Oct 30 06:02:01.785988 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.785969 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5749bffc96-tr2hn" Oct 30 06:02:01.804101 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.804046 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 06:02:01.816166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.816144 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-52xdc\"" Oct 30 06:02:01.816166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.816153 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 06:02:01.816346 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.816242 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 06:02:01.816429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.816415 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 06:02:01.816497 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.816447 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 06:02:01.829310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.829290 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5749bffc96-tr2hn"] Oct 30 06:02:01.853416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.853391 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c87696f-3650-470f-9ddf-2d1137017d59-webhook-cert\") pod \"maas-controller-5749bffc96-tr2hn\" (UID: \"9c87696f-3650-470f-9ddf-2d1137017d59\") " pod="opendatahub/maas-controller-5749bffc96-tr2hn" Oct 30 06:02:01.853518 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.853417 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c87696f-3650-470f-9ddf-2d1137017d59-metrics-certs\") pod \"maas-controller-5749bffc96-tr2hn\" (UID: \"9c87696f-3650-470f-9ddf-2d1137017d59\") " pod="opendatahub/maas-controller-5749bffc96-tr2hn" Oct 30 06:02:01.853518 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.853513 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4hpc\" (UniqueName: \"kubernetes.io/projected/9c87696f-3650-470f-9ddf-2d1137017d59-kube-api-access-c4hpc\") pod \"maas-controller-5749bffc96-tr2hn\" (UID: \"9c87696f-3650-470f-9ddf-2d1137017d59\") " pod="opendatahub/maas-controller-5749bffc96-tr2hn" Oct 30 06:02:01.954252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.954198 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c87696f-3650-470f-9ddf-2d1137017d59-webhook-cert\") pod \"maas-controller-5749bffc96-tr2hn\" (UID: \"9c87696f-3650-470f-9ddf-2d1137017d59\") " pod="opendatahub/maas-controller-5749bffc96-tr2hn" Oct 30 06:02:01.954252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.954255 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c87696f-3650-470f-9ddf-2d1137017d59-metrics-certs\") pod \"maas-controller-5749bffc96-tr2hn\" (UID: \"9c87696f-3650-470f-9ddf-2d1137017d59\") " pod="opendatahub/maas-controller-5749bffc96-tr2hn" Oct 30 06:02:01.954449 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.954307 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4hpc\" (UniqueName: \"kubernetes.io/projected/9c87696f-3650-470f-9ddf-2d1137017d59-kube-api-access-c4hpc\") pod \"maas-controller-5749bffc96-tr2hn\" (UID: \"9c87696f-3650-470f-9ddf-2d1137017d59\") " pod="opendatahub/maas-controller-5749bffc96-tr2hn" Oct 30 06:02:01.956785 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.956760 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c87696f-3650-470f-9ddf-2d1137017d59-webhook-cert\") pod \"maas-controller-5749bffc96-tr2hn\" (UID: \"9c87696f-3650-470f-9ddf-2d1137017d59\") " pod="opendatahub/maas-controller-5749bffc96-tr2hn" Oct 30 06:02:01.956898 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.956885 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c87696f-3650-470f-9ddf-2d1137017d59-metrics-certs\") pod \"maas-controller-5749bffc96-tr2hn\" (UID: \"9c87696f-3650-470f-9ddf-2d1137017d59\") " pod="opendatahub/maas-controller-5749bffc96-tr2hn" Oct 30 06:02:01.979091 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:01.979056 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4hpc\" (UniqueName: \"kubernetes.io/projected/9c87696f-3650-470f-9ddf-2d1137017d59-kube-api-access-c4hpc\") pod \"maas-controller-5749bffc96-tr2hn\" (UID: \"9c87696f-3650-470f-9ddf-2d1137017d59\") " pod="opendatahub/maas-controller-5749bffc96-tr2hn" Oct 30 06:02:02.099126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:02.099047 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5749bffc96-tr2hn" Oct 30 06:02:02.413286 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:02.413257 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5749bffc96-tr2hn"] Oct 30 06:02:02.413970 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:02:02.413941 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c87696f_3650_470f_9ddf_2d1137017d59.slice/crio-f37dcd10761459fda76887f81853e2346d11823d9d27e66087ac54bc95085855 WatchSource:0}: Error finding container f37dcd10761459fda76887f81853e2346d11823d9d27e66087ac54bc95085855: Status 404 returned error can't find the container with id f37dcd10761459fda76887f81853e2346d11823d9d27e66087ac54bc95085855 Oct 30 06:02:02.549404 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:02.549367 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5749bffc96-tr2hn" event={"ID":"9c87696f-3650-470f-9ddf-2d1137017d59","Type":"ContainerStarted","Data":"f37dcd10761459fda76887f81853e2346d11823d9d27e66087ac54bc95085855"} Oct 30 06:02:04.145524 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:02:04.145433 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:02:04.574975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:04.574897 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:04.751960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:04.751922 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:05.564662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:05.564621 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5749bffc96-tr2hn" event={"ID":"9c87696f-3650-470f-9ddf-2d1137017d59","Type":"ContainerStarted","Data":"209d4a7a75ef9f03477b20553285fc731e2a519cea74bb4b5d9987ad20f2babb"} Oct 30 06:02:05.565163 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:05.564730 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5749bffc96-tr2hn" Oct 30 06:02:05.615511 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:05.615468 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5749bffc96-tr2hn" podStartSLOduration=2.331434713 podStartE2EDuration="4.615455243s" totalImagesPullingTime="2.28402053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:05.613848926 +0000 UTC m=+492.122868814" watchObservedRunningTime="2026-10-30 06:02:05.615455243 +0000 UTC m=+492.124475130" Oct 30 06:02:05.967951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:05.967915 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb"] Oct 30 06:02:05.971489 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:05.971468 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:05.992772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:05.992749 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 06:02:05.992878 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:05.992862 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-zg5lg\"" Oct 30 06:02:06.006940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.006915 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb"] Oct 30 06:02:06.014344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.014323 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 06:02:06.014508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.014491 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 06:02:06.077573 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.077540 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:06.091363 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.091340 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmdh7\" (UniqueName: \"kubernetes.io/projected/4d8da3c6-77af-4a68-aa92-89594aaa5037-kube-api-access-wmdh7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wmwrb\" (UID: \"4d8da3c6-77af-4a68-aa92-89594aaa5037\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:06.091426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.091384 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4d8da3c6-77af-4a68-aa92-89594aaa5037-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wmwrb\" (UID: \"4d8da3c6-77af-4a68-aa92-89594aaa5037\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:06.091531 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.091517 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d8da3c6-77af-4a68-aa92-89594aaa5037-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wmwrb\" (UID: \"4d8da3c6-77af-4a68-aa92-89594aaa5037\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:06.091567 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.091550 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4d8da3c6-77af-4a68-aa92-89594aaa5037-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wmwrb\" (UID: \"4d8da3c6-77af-4a68-aa92-89594aaa5037\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:06.192490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.192455 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d8da3c6-77af-4a68-aa92-89594aaa5037-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wmwrb\" (UID: \"4d8da3c6-77af-4a68-aa92-89594aaa5037\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:06.192613 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.192498 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4d8da3c6-77af-4a68-aa92-89594aaa5037-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wmwrb\" (UID: \"4d8da3c6-77af-4a68-aa92-89594aaa5037\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:06.192664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.192630 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmdh7\" (UniqueName: \"kubernetes.io/projected/4d8da3c6-77af-4a68-aa92-89594aaa5037-kube-api-access-wmdh7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wmwrb\" (UID: \"4d8da3c6-77af-4a68-aa92-89594aaa5037\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:06.192710 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.192693 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4d8da3c6-77af-4a68-aa92-89594aaa5037-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wmwrb\" (UID: \"4d8da3c6-77af-4a68-aa92-89594aaa5037\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:06.192802 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:02:06.192778 6999 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 30 06:02:06.192882 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:02:06.192871 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d8da3c6-77af-4a68-aa92-89594aaa5037-tls-cert podName:4d8da3c6-77af-4a68-aa92-89594aaa5037 nodeName:}" failed. No retries permitted until 2026-10-30 06:02:06.692849529 +0000 UTC m=+493.201869411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/4d8da3c6-77af-4a68-aa92-89594aaa5037-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" (UID: "4d8da3c6-77af-4a68-aa92-89594aaa5037") : secret "usage-logs-tenancy-proxy-tls" not found Oct 30 06:02:06.193151 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.193135 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d8da3c6-77af-4a68-aa92-89594aaa5037-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wmwrb\" (UID: \"4d8da3c6-77af-4a68-aa92-89594aaa5037\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:06.193368 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.193350 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4d8da3c6-77af-4a68-aa92-89594aaa5037-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wmwrb\" (UID: \"4d8da3c6-77af-4a68-aa92-89594aaa5037\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:06.221949 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.221892 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmdh7\" (UniqueName: \"kubernetes.io/projected/4d8da3c6-77af-4a68-aa92-89594aaa5037-kube-api-access-wmdh7\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wmwrb\" (UID: \"4d8da3c6-77af-4a68-aa92-89594aaa5037\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:06.329899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.329860 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:06.699071 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.699049 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4d8da3c6-77af-4a68-aa92-89594aaa5037-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wmwrb\" (UID: \"4d8da3c6-77af-4a68-aa92-89594aaa5037\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:06.701696 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.701672 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4d8da3c6-77af-4a68-aa92-89594aaa5037-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-wmwrb\" (UID: \"4d8da3c6-77af-4a68-aa92-89594aaa5037\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:06.884841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:06.884814 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:07.184468 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:07.184446 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb"] Oct 30 06:02:07.185902 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:02:07.185881 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8da3c6_77af_4a68_aa92_89594aaa5037.slice/crio-e9731acd66514fbcb8803f6fbfe23963ea9b7cdb878ddef94071338c3d9bea8b WatchSource:0}: Error finding container e9731acd66514fbcb8803f6fbfe23963ea9b7cdb878ddef94071338c3d9bea8b: Status 404 returned error can't find the container with id e9731acd66514fbcb8803f6fbfe23963ea9b7cdb878ddef94071338c3d9bea8b Oct 30 06:02:07.576026 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:07.575952 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" event={"ID":"4d8da3c6-77af-4a68-aa92-89594aaa5037","Type":"ContainerStarted","Data":"e9731acd66514fbcb8803f6fbfe23963ea9b7cdb878ddef94071338c3d9bea8b"} Oct 30 06:02:12.419407 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:12.419352 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:12.831327 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:12.831110 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:16.571919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:16.571884 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5749bffc96-tr2hn" Oct 30 06:02:17.535335 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:17.535272 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:18.633643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:18.633613 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" event={"ID":"4d8da3c6-77af-4a68-aa92-89594aaa5037","Type":"ContainerStarted","Data":"a88a84ab635680f1983846beb23a59eb3f70613ff1832bb990d9c52da03b3c91"} Oct 30 06:02:18.634000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:18.633689 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:18.681375 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:18.681331 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" podStartSLOduration=2.687463221 podStartE2EDuration="13.681317941s" totalImagesPullingTime="10.99385472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:18.680672018 +0000 UTC m=+505.189691915" watchObservedRunningTime="2026-10-30 06:02:18.681317941 +0000 UTC m=+505.190337819" Oct 30 06:02:19.625285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:19.625251 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:19.825077 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:19.825045 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:19.925906 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:19.925833 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-dd578"] Oct 30 06:02:19.929562 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:19.929547 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-dd578" Oct 30 06:02:19.938929 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:19.938905 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2d51b73-b344-44d8-8fbf-84b50e45cbe5-cert\") pod \"kserve-controller-manager-776cc66d47-dd578\" (UID: \"d2d51b73-b344-44d8-8fbf-84b50e45cbe5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dd578" Oct 30 06:02:19.939057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:19.939038 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jrr6\" (UniqueName: \"kubernetes.io/projected/d2d51b73-b344-44d8-8fbf-84b50e45cbe5-kube-api-access-8jrr6\") pod \"kserve-controller-manager-776cc66d47-dd578\" (UID: \"d2d51b73-b344-44d8-8fbf-84b50e45cbe5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dd578" Oct 30 06:02:19.960429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:19.960401 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 06:02:19.960535 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:19.960435 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vkv4m\"" Oct 30 06:02:19.992626 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:19.992603 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-dd578"] Oct 30 06:02:20.039994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.039969 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jrr6\" (UniqueName: \"kubernetes.io/projected/d2d51b73-b344-44d8-8fbf-84b50e45cbe5-kube-api-access-8jrr6\") pod \"kserve-controller-manager-776cc66d47-dd578\" (UID: \"d2d51b73-b344-44d8-8fbf-84b50e45cbe5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dd578" Oct 30 06:02:20.040119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.040003 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2d51b73-b344-44d8-8fbf-84b50e45cbe5-cert\") pod \"kserve-controller-manager-776cc66d47-dd578\" (UID: \"d2d51b73-b344-44d8-8fbf-84b50e45cbe5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dd578" Oct 30 06:02:20.042630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.042603 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2d51b73-b344-44d8-8fbf-84b50e45cbe5-cert\") pod \"kserve-controller-manager-776cc66d47-dd578\" (UID: \"d2d51b73-b344-44d8-8fbf-84b50e45cbe5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dd578" Oct 30 06:02:20.080191 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.080169 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jrr6\" (UniqueName: \"kubernetes.io/projected/d2d51b73-b344-44d8-8fbf-84b50e45cbe5-kube-api-access-8jrr6\") pod \"kserve-controller-manager-776cc66d47-dd578\" (UID: \"d2d51b73-b344-44d8-8fbf-84b50e45cbe5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-dd578" Oct 30 06:02:20.092378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.092354 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw"] Oct 30 06:02:20.098337 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.098321 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-wm62c"] Oct 30 06:02:20.098449 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.098433 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" Oct 30 06:02:20.104898 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.104886 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-wm62c" Oct 30 06:02:20.122877 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.122862 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-l64cd\"" Oct 30 06:02:20.122877 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.122861 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 06:02:20.135065 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.135043 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:20.138495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.138470 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw"] Oct 30 06:02:20.139481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.139462 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-wm62c"] Oct 30 06:02:20.141527 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.141503 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72rg7\" (UniqueName: \"kubernetes.io/projected/683de426-d0ef-4eb9-8c8f-3ca3e2a066a4-kube-api-access-72rg7\") pod \"llmisvc-controller-manager-869d5dcc5d-2lbgw\" (UID: \"683de426-d0ef-4eb9-8c8f-3ca3e2a066a4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" Oct 30 06:02:20.141684 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.141661 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/683de426-d0ef-4eb9-8c8f-3ca3e2a066a4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2lbgw\" (UID: \"683de426-d0ef-4eb9-8c8f-3ca3e2a066a4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" Oct 30 06:02:20.190125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.190072 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-kjczm"] Oct 30 06:02:20.204313 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.204287 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-kjczm" Oct 30 06:02:20.218108 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.218088 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-kjczm"] Oct 30 06:02:20.219679 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.219661 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-zqkk4\"" Oct 30 06:02:20.219918 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.219759 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 06:02:20.241537 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.241513 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-dd578" Oct 30 06:02:20.242807 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.242785 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c65427c6-ca18-462d-b777-c52b0b195c9c-cert\") pod \"odh-model-controller-769875db56-kjczm\" (UID: \"c65427c6-ca18-462d-b777-c52b0b195c9c\") " pod="opendatahub/odh-model-controller-769875db56-kjczm" Oct 30 06:02:20.242913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.242823 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72rg7\" (UniqueName: \"kubernetes.io/projected/683de426-d0ef-4eb9-8c8f-3ca3e2a066a4-kube-api-access-72rg7\") pod \"llmisvc-controller-manager-869d5dcc5d-2lbgw\" (UID: \"683de426-d0ef-4eb9-8c8f-3ca3e2a066a4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" Oct 30 06:02:20.242913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.242846 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/683de426-d0ef-4eb9-8c8f-3ca3e2a066a4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2lbgw\" (UID: \"683de426-d0ef-4eb9-8c8f-3ca3e2a066a4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" Oct 30 06:02:20.243004 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.242916 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3d6cb4-b70b-4d68-aca8-53fdb36f017d-tls-certs\") pod \"model-serving-api-75bd89fdfc-wm62c\" (UID: \"0e3d6cb4-b70b-4d68-aca8-53fdb36f017d\") " pod="opendatahub/model-serving-api-75bd89fdfc-wm62c" Oct 30 06:02:20.243004 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.242939 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c65427c6-ca18-462d-b777-c52b0b195c9c-metrics-certs\") pod \"odh-model-controller-769875db56-kjczm\" (UID: \"c65427c6-ca18-462d-b777-c52b0b195c9c\") " pod="opendatahub/odh-model-controller-769875db56-kjczm" Oct 30 06:02:20.243004 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.242992 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgkqj\" (UniqueName: \"kubernetes.io/projected/c65427c6-ca18-462d-b777-c52b0b195c9c-kube-api-access-dgkqj\") pod \"odh-model-controller-769875db56-kjczm\" (UID: \"c65427c6-ca18-462d-b777-c52b0b195c9c\") " pod="opendatahub/odh-model-controller-769875db56-kjczm" Oct 30 06:02:20.243106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.243042 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p9rc\" (UniqueName: \"kubernetes.io/projected/0e3d6cb4-b70b-4d68-aca8-53fdb36f017d-kube-api-access-6p9rc\") pod \"model-serving-api-75bd89fdfc-wm62c\" (UID: \"0e3d6cb4-b70b-4d68-aca8-53fdb36f017d\") " pod="opendatahub/model-serving-api-75bd89fdfc-wm62c" Oct 30 06:02:20.245491 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.245473 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/683de426-d0ef-4eb9-8c8f-3ca3e2a066a4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2lbgw\" (UID: \"683de426-d0ef-4eb9-8c8f-3ca3e2a066a4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" Oct 30 06:02:20.266623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.266589 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72rg7\" (UniqueName: \"kubernetes.io/projected/683de426-d0ef-4eb9-8c8f-3ca3e2a066a4-kube-api-access-72rg7\") pod \"llmisvc-controller-manager-869d5dcc5d-2lbgw\" (UID: \"683de426-d0ef-4eb9-8c8f-3ca3e2a066a4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" Oct 30 06:02:20.314075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.314047 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:20.319326 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.319304 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 06:02:20.319457 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.319356 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-rxklx\"" Oct 30 06:02:20.319457 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.319362 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 06:02:20.343800 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.343777 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p9rc\" (UniqueName: \"kubernetes.io/projected/0e3d6cb4-b70b-4d68-aca8-53fdb36f017d-kube-api-access-6p9rc\") pod \"model-serving-api-75bd89fdfc-wm62c\" (UID: \"0e3d6cb4-b70b-4d68-aca8-53fdb36f017d\") " pod="opendatahub/model-serving-api-75bd89fdfc-wm62c" Oct 30 06:02:20.343927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.343812 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c65427c6-ca18-462d-b777-c52b0b195c9c-cert\") pod \"odh-model-controller-769875db56-kjczm\" (UID: \"c65427c6-ca18-462d-b777-c52b0b195c9c\") " pod="opendatahub/odh-model-controller-769875db56-kjczm" Oct 30 06:02:20.343927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.343897 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3d6cb4-b70b-4d68-aca8-53fdb36f017d-tls-certs\") pod \"model-serving-api-75bd89fdfc-wm62c\" (UID: \"0e3d6cb4-b70b-4d68-aca8-53fdb36f017d\") " pod="opendatahub/model-serving-api-75bd89fdfc-wm62c" Oct 30 06:02:20.343927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.343921 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c65427c6-ca18-462d-b777-c52b0b195c9c-metrics-certs\") pod \"odh-model-controller-769875db56-kjczm\" (UID: \"c65427c6-ca18-462d-b777-c52b0b195c9c\") " pod="opendatahub/odh-model-controller-769875db56-kjczm" Oct 30 06:02:20.344101 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.343958 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgkqj\" (UniqueName: \"kubernetes.io/projected/c65427c6-ca18-462d-b777-c52b0b195c9c-kube-api-access-dgkqj\") pod \"odh-model-controller-769875db56-kjczm\" (UID: \"c65427c6-ca18-462d-b777-c52b0b195c9c\") " pod="opendatahub/odh-model-controller-769875db56-kjczm" Oct 30 06:02:20.346747 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.346726 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c65427c6-ca18-462d-b777-c52b0b195c9c-metrics-certs\") pod \"odh-model-controller-769875db56-kjczm\" (UID: \"c65427c6-ca18-462d-b777-c52b0b195c9c\") " pod="opendatahub/odh-model-controller-769875db56-kjczm" Oct 30 06:02:20.347024 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.347005 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c65427c6-ca18-462d-b777-c52b0b195c9c-cert\") pod \"odh-model-controller-769875db56-kjczm\" (UID: \"c65427c6-ca18-462d-b777-c52b0b195c9c\") " pod="opendatahub/odh-model-controller-769875db56-kjczm" Oct 30 06:02:20.347109 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.347009 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e3d6cb4-b70b-4d68-aca8-53fdb36f017d-tls-certs\") pod \"model-serving-api-75bd89fdfc-wm62c\" (UID: \"0e3d6cb4-b70b-4d68-aca8-53fdb36f017d\") " pod="opendatahub/model-serving-api-75bd89fdfc-wm62c" Oct 30 06:02:20.373104 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.373085 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p9rc\" (UniqueName: \"kubernetes.io/projected/0e3d6cb4-b70b-4d68-aca8-53fdb36f017d-kube-api-access-6p9rc\") pod \"model-serving-api-75bd89fdfc-wm62c\" (UID: \"0e3d6cb4-b70b-4d68-aca8-53fdb36f017d\") " pod="opendatahub/model-serving-api-75bd89fdfc-wm62c" Oct 30 06:02:20.373202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.373186 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgkqj\" (UniqueName: \"kubernetes.io/projected/c65427c6-ca18-462d-b777-c52b0b195c9c-kube-api-access-dgkqj\") pod \"odh-model-controller-769875db56-kjczm\" (UID: \"c65427c6-ca18-462d-b777-c52b0b195c9c\") " pod="opendatahub/odh-model-controller-769875db56-kjczm" Oct 30 06:02:20.416383 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.416359 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" Oct 30 06:02:20.427495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.427465 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-wm62c" Oct 30 06:02:20.517745 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.517560 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-kjczm" Oct 30 06:02:20.545109 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.545081 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-dd578"] Oct 30 06:02:20.546537 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:02:20.546508 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d51b73_b344_44d8_8fbf_84b50e45cbe5.slice/crio-ee2ffdb8e266cf56e5f01cc0ccca7d98a83b14ac0fcec8ba81c21cef457edb01 WatchSource:0}: Error finding container ee2ffdb8e266cf56e5f01cc0ccca7d98a83b14ac0fcec8ba81c21cef457edb01: Status 404 returned error can't find the container with id ee2ffdb8e266cf56e5f01cc0ccca7d98a83b14ac0fcec8ba81c21cef457edb01 Oct 30 06:02:20.569939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.569912 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:20.643167 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.643140 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-dd578" event={"ID":"d2d51b73-b344-44d8-8fbf-84b50e45cbe5","Type":"ContainerStarted","Data":"ee2ffdb8e266cf56e5f01cc0ccca7d98a83b14ac0fcec8ba81c21cef457edb01"} Oct 30 06:02:20.735984 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.735958 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw"] Oct 30 06:02:20.736852 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:02:20.736824 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod683de426_d0ef_4eb9_8c8f_3ca3e2a066a4.slice/crio-4f5404d48084c43b87b322330db9837806775df6600aec37297e1d34d2aff55c WatchSource:0}: Error finding container 4f5404d48084c43b87b322330db9837806775df6600aec37297e1d34d2aff55c: Status 404 returned error can't find the container with id 4f5404d48084c43b87b322330db9837806775df6600aec37297e1d34d2aff55c Oct 30 06:02:20.760129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.759948 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-wm62c"] Oct 30 06:02:20.760937 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:02:20.760914 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e3d6cb4_b70b_4d68_aca8_53fdb36f017d.slice/crio-bb3b11ee06d92110e9d55db8b4cd13630be72368342d56b4c002f8e5d0f39d71 WatchSource:0}: Error finding container bb3b11ee06d92110e9d55db8b4cd13630be72368342d56b4c002f8e5d0f39d71: Status 404 returned error can't find the container with id bb3b11ee06d92110e9d55db8b4cd13630be72368342d56b4c002f8e5d0f39d71 Oct 30 06:02:20.819508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:20.819483 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-kjczm"] Oct 30 06:02:20.820566 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:02:20.820541 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65427c6_ca18_462d_b777_c52b0b195c9c.slice/crio-999a463c7269f49e15c8d24b90bb6885fd0487a6932286fa9bf8d82f264d4729 WatchSource:0}: Error finding container 999a463c7269f49e15c8d24b90bb6885fd0487a6932286fa9bf8d82f264d4729: Status 404 returned error can't find the container with id 999a463c7269f49e15c8d24b90bb6885fd0487a6932286fa9bf8d82f264d4729 Oct 30 06:02:21.656572 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:21.656532 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-kjczm" event={"ID":"c65427c6-ca18-462d-b777-c52b0b195c9c","Type":"ContainerStarted","Data":"999a463c7269f49e15c8d24b90bb6885fd0487a6932286fa9bf8d82f264d4729"} Oct 30 06:02:21.658381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:21.658353 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-wm62c" event={"ID":"0e3d6cb4-b70b-4d68-aca8-53fdb36f017d","Type":"ContainerStarted","Data":"bb3b11ee06d92110e9d55db8b4cd13630be72368342d56b4c002f8e5d0f39d71"} Oct 30 06:02:21.660351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:21.660327 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" event={"ID":"683de426-d0ef-4eb9-8c8f-3ca3e2a066a4","Type":"ContainerStarted","Data":"4f5404d48084c43b87b322330db9837806775df6600aec37297e1d34d2aff55c"} Oct 30 06:02:22.510199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:22.510162 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:22.718609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:22.718528 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:22.884247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:22.884194 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:23.033686 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:23.033640 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:23.674800 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:23.674768 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-wm62c" event={"ID":"0e3d6cb4-b70b-4d68-aca8-53fdb36f017d","Type":"ContainerStarted","Data":"2c7dd4b49c84b8067e00fdc3c002f5751c3643f42c328fb0e71617e33b9bd0a9"} Oct 30 06:02:23.675056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:23.675038 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-wm62c" Oct 30 06:02:23.724201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:23.723572 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-wm62c" podStartSLOduration=1.125113109 podStartE2EDuration="3.723540468s" totalImagesPullingTime="2.598427359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:23.721473254 +0000 UTC m=+510.230493155" watchObservedRunningTime="2026-10-30 06:02:23.723540468 +0000 UTC m=+510.232560356" Oct 30 06:02:24.681087 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:24.680985 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-dd578" event={"ID":"d2d51b73-b344-44d8-8fbf-84b50e45cbe5","Type":"ContainerStarted","Data":"550b132df200facad236b0f06d9f0b7ff0fcebe4154a3c79a59a080ed3b742c9"} Oct 30 06:02:24.681562 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:24.681534 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-dd578" Oct 30 06:02:24.682740 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:24.682716 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-kjczm" event={"ID":"c65427c6-ca18-462d-b777-c52b0b195c9c","Type":"ContainerStarted","Data":"caf5c0c8d5ecff18664164e6286f916e302b20072d5074c9f9fd8a7917193dac"} Oct 30 06:02:24.682867 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:24.682850 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-kjczm" Oct 30 06:02:24.684526 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:24.684507 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" event={"ID":"683de426-d0ef-4eb9-8c8f-3ca3e2a066a4","Type":"ContainerStarted","Data":"0aa7ec7d9c4c6410b5d090b1cf089d03310c5e031aee2f6194743f2cf5038162"} Oct 30 06:02:24.684642 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:24.684618 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" Oct 30 06:02:24.722871 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:24.722835 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-dd578" podStartSLOduration=2.627501385 podStartE2EDuration="5.722823888s" totalImagesPullingTime="3.095322503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:24.722421246 +0000 UTC m=+511.231441139" watchObservedRunningTime="2026-10-30 06:02:24.722823888 +0000 UTC m=+511.231843769" Oct 30 06:02:24.773767 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:24.773715 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-kjczm" podStartSLOduration=1.163824691 podStartE2EDuration="4.773699427s" totalImagesPullingTime="3.609874736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:24.771146914 +0000 UTC m=+511.280166802" watchObservedRunningTime="2026-10-30 06:02:24.773699427 +0000 UTC m=+511.282719315" Oct 30 06:02:24.814459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:24.814412 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" podStartSLOduration=2.885430434 podStartE2EDuration="5.81439902s" totalImagesPullingTime="2.928968586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:24.812866255 +0000 UTC m=+511.321886142" watchObservedRunningTime="2026-10-30 06:02:24.81439902 +0000 UTC m=+511.323418900" Oct 30 06:02:26.223617 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:26.223585 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:26.402467 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:26.402424 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:29.644119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:29.644094 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-wmwrb" Oct 30 06:02:31.922469 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:31.922435 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:34.042476 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:34.042444 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:34.140763 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:02:34.140729 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:02:34.191642 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:34.191616 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:34.688929 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:34.688905 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-wm62c" Oct 30 06:02:35.584553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:35.584523 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:35.691373 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:35.691331 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-kjczm" Oct 30 06:02:35.731963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:35.731932 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:35.894560 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:35.894532 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:45.850276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:45.850235 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:45.983595 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:45.983557 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:46.123587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.123563 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rmd8t"] Oct 30 06:02:46.384236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.384144 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rmd8t"] Oct 30 06:02:46.384236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.384177 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rmd8t"] Oct 30 06:02:46.384449 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.384275 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rmd8t" Oct 30 06:02:46.409226 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.409189 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rmd8t" Oct 30 06:02:46.447484 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.447465 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-5ls5l"] Oct 30 06:02:46.454884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.454867 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-5ls5l" Oct 30 06:02:46.497861 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.497843 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 06:02:46.521760 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.521737 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-5ls5l"] Oct 30 06:02:46.576628 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.576609 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-5ls5l"] Oct 30 06:02:46.576813 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:02:46.576797 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xfnr4 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-5ls5l" podUID="29e3b9f4-d540-4717-82b4-cd874ce4db15" Oct 30 06:02:46.585864 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.585841 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29e3b9f4-d540-4717-82b4-cd874ce4db15-tls-cert\") pod \"authorino-6889cbd8c4-5ls5l\" (UID: \"29e3b9f4-d540-4717-82b4-cd874ce4db15\") " pod="kuadrant-system/authorino-6889cbd8c4-5ls5l" Oct 30 06:02:46.585957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.585875 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfnr4\" (UniqueName: \"kubernetes.io/projected/29e3b9f4-d540-4717-82b4-cd874ce4db15-kube-api-access-xfnr4\") pod \"authorino-6889cbd8c4-5ls5l\" (UID: \"29e3b9f4-d540-4717-82b4-cd874ce4db15\") " pod="kuadrant-system/authorino-6889cbd8c4-5ls5l" Oct 30 06:02:46.609707 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.609680 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-674cd869b7-wvxv5"] Oct 30 06:02:46.651242 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.651159 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674cd869b7-wvxv5"] Oct 30 06:02:46.651345 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.651260 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674cd869b7-wvxv5" Oct 30 06:02:46.686918 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.686894 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29e3b9f4-d540-4717-82b4-cd874ce4db15-tls-cert\") pod \"authorino-6889cbd8c4-5ls5l\" (UID: \"29e3b9f4-d540-4717-82b4-cd874ce4db15\") " pod="kuadrant-system/authorino-6889cbd8c4-5ls5l" Oct 30 06:02:46.687031 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.686929 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfnr4\" (UniqueName: \"kubernetes.io/projected/29e3b9f4-d540-4717-82b4-cd874ce4db15-kube-api-access-xfnr4\") pod \"authorino-6889cbd8c4-5ls5l\" (UID: \"29e3b9f4-d540-4717-82b4-cd874ce4db15\") " pod="kuadrant-system/authorino-6889cbd8c4-5ls5l" Oct 30 06:02:46.689527 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.689509 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29e3b9f4-d540-4717-82b4-cd874ce4db15-tls-cert\") pod \"authorino-6889cbd8c4-5ls5l\" (UID: \"29e3b9f4-d540-4717-82b4-cd874ce4db15\") " pod="kuadrant-system/authorino-6889cbd8c4-5ls5l" Oct 30 06:02:46.710290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.710266 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfnr4\" (UniqueName: \"kubernetes.io/projected/29e3b9f4-d540-4717-82b4-cd874ce4db15-kube-api-access-xfnr4\") pod \"authorino-6889cbd8c4-5ls5l\" (UID: \"29e3b9f4-d540-4717-82b4-cd874ce4db15\") " pod="kuadrant-system/authorino-6889cbd8c4-5ls5l" Oct 30 06:02:46.785504 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.785481 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-5ls5l" Oct 30 06:02:46.785601 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.785481 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-rmd8t" Oct 30 06:02:46.787304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.787284 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d90f37f-b29a-4577-ae53-7e8ca34bda9a-tls-cert\") pod \"authorino-674cd869b7-wvxv5\" (UID: \"8d90f37f-b29a-4577-ae53-7e8ca34bda9a\") " pod="kuadrant-system/authorino-674cd869b7-wvxv5" Oct 30 06:02:46.787397 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.787345 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85v28\" (UniqueName: \"kubernetes.io/projected/8d90f37f-b29a-4577-ae53-7e8ca34bda9a-kube-api-access-85v28\") pod \"authorino-674cd869b7-wvxv5\" (UID: \"8d90f37f-b29a-4577-ae53-7e8ca34bda9a\") " pod="kuadrant-system/authorino-674cd869b7-wvxv5" Oct 30 06:02:46.811340 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.811319 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-5ls5l" Oct 30 06:02:46.888228 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.888190 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e3b9f4-d540-4717-82b4-cd874ce4db15-kube-api-access-xfnr4\" (UniqueName: \"kubernetes.io/projected/29e3b9f4-d540-4717-82b4-cd874ce4db15-kube-api-access-xfnr4\") pod \"29e3b9f4-d540-4717-82b4-cd874ce4db15\" (UID: \"29e3b9f4-d540-4717-82b4-cd874ce4db15\") " Oct 30 06:02:46.888509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.888310 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29e3b9f4-d540-4717-82b4-cd874ce4db15-tls-cert\" (UniqueName: \"kubernetes.io/secret/29e3b9f4-d540-4717-82b4-cd874ce4db15-tls-cert\") pod \"29e3b9f4-d540-4717-82b4-cd874ce4db15\" (UID: \"29e3b9f4-d540-4717-82b4-cd874ce4db15\") " Oct 30 06:02:46.888509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.888458 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85v28\" (UniqueName: \"kubernetes.io/projected/8d90f37f-b29a-4577-ae53-7e8ca34bda9a-kube-api-access-85v28\") pod \"authorino-674cd869b7-wvxv5\" (UID: \"8d90f37f-b29a-4577-ae53-7e8ca34bda9a\") " pod="kuadrant-system/authorino-674cd869b7-wvxv5" Oct 30 06:02:46.888602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.888549 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d90f37f-b29a-4577-ae53-7e8ca34bda9a-tls-cert\") pod \"authorino-674cd869b7-wvxv5\" (UID: \"8d90f37f-b29a-4577-ae53-7e8ca34bda9a\") " pod="kuadrant-system/authorino-674cd869b7-wvxv5" Oct 30 06:02:46.890481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.890455 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e3b9f4-d540-4717-82b4-cd874ce4db15-tls-cert" pod "29e3b9f4-d540-4717-82b4-cd874ce4db15" (UID: "29e3b9f4-d540-4717-82b4-cd874ce4db15"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:46.890567 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.890478 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e3b9f4-d540-4717-82b4-cd874ce4db15-kube-api-access-xfnr4" pod "29e3b9f4-d540-4717-82b4-cd874ce4db15" (UID: "29e3b9f4-d540-4717-82b4-cd874ce4db15"). InnerVolumeSpecName "kube-api-access-xfnr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:46.890943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.890928 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d90f37f-b29a-4577-ae53-7e8ca34bda9a-tls-cert\") pod \"authorino-674cd869b7-wvxv5\" (UID: \"8d90f37f-b29a-4577-ae53-7e8ca34bda9a\") " pod="kuadrant-system/authorino-674cd869b7-wvxv5" Oct 30 06:02:46.912539 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.912468 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85v28\" (UniqueName: \"kubernetes.io/projected/8d90f37f-b29a-4577-ae53-7e8ca34bda9a-kube-api-access-85v28\") pod \"authorino-674cd869b7-wvxv5\" (UID: \"8d90f37f-b29a-4577-ae53-7e8ca34bda9a\") " pod="kuadrant-system/authorino-674cd869b7-wvxv5" Oct 30 06:02:46.919579 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.919552 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rmd8t"] Oct 30 06:02:46.921635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.921614 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-rmd8t"] Oct 30 06:02:46.982001 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.981979 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674cd869b7-wvxv5" Oct 30 06:02:46.989402 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.989382 6999 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29e3b9f4-d540-4717-82b4-cd874ce4db15-tls-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:02:46.989491 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:46.989403 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfnr4\" (UniqueName: \"kubernetes.io/projected/29e3b9f4-d540-4717-82b4-cd874ce4db15-kube-api-access-xfnr4\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:02:47.294839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:47.294815 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674cd869b7-wvxv5"] Oct 30 06:02:47.298357 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:02:47.298276 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d90f37f_b29a_4577_ae53_7e8ca34bda9a.slice/crio-586a6b5edcdcbd43fa7e59cb7b7042c8f29cc10bce0d231db9853b241ed688e9 WatchSource:0}: Error finding container 586a6b5edcdcbd43fa7e59cb7b7042c8f29cc10bce0d231db9853b241ed688e9: Status 404 returned error can't find the container with id 586a6b5edcdcbd43fa7e59cb7b7042c8f29cc10bce0d231db9853b241ed688e9 Oct 30 06:02:47.791423 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:47.791388 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674cd869b7-wvxv5" event={"ID":"8d90f37f-b29a-4577-ae53-7e8ca34bda9a","Type":"ContainerStarted","Data":"586a6b5edcdcbd43fa7e59cb7b7042c8f29cc10bce0d231db9853b241ed688e9"} Oct 30 06:02:47.791603 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:47.791403 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-5ls5l" Oct 30 06:02:47.869330 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:47.869265 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-5ls5l"] Oct 30 06:02:47.877884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:47.877858 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-5ls5l"] Oct 30 06:02:48.109503 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:48.109473 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:48.498972 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:48.498936 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:02:48.801744 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:48.801700 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674cd869b7-wvxv5" event={"ID":"8d90f37f-b29a-4577-ae53-7e8ca34bda9a","Type":"ContainerStarted","Data":"289d0448449c9a055b6c8d906d51cec5bc5aee92b9f9f53161b35d53bff262c4"} Oct 30 06:02:48.860036 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:48.859969 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-674cd869b7-wvxv5" podStartSLOduration=1.8520252849999999 podStartE2EDuration="2.859948477s" totalImagesPullingTime="1.007923192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:48.856741094 +0000 UTC m=+535.365760981" watchObservedRunningTime="2026-10-30 06:02:48.859948477 +0000 UTC m=+535.368968364" Oct 30 06:02:48.909827 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:48.909769 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bqddx"] Oct 30 06:02:48.910319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:48.910026 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-bqddx" podUID="f8f10885-315c-4312-a86f-06d595294e63" containerName="authorino" containerID="cri-o://87b4b53a07c2d24ed7aaa44f63abe62fc0e745c3052dab0f4ea2ca2984dc60c4" gracePeriod=30 Oct 30 06:02:49.209678 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:49.209650 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bqddx" Oct 30 06:02:49.311420 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:49.311386 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f10885-315c-4312-a86f-06d595294e63-kube-api-access-lg62x\" (UniqueName: \"kubernetes.io/projected/f8f10885-315c-4312-a86f-06d595294e63-kube-api-access-lg62x\") pod \"f8f10885-315c-4312-a86f-06d595294e63\" (UID: \"f8f10885-315c-4312-a86f-06d595294e63\") " Oct 30 06:02:49.313387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:49.313355 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f10885-315c-4312-a86f-06d595294e63-kube-api-access-lg62x" pod "f8f10885-315c-4312-a86f-06d595294e63" (UID: "f8f10885-315c-4312-a86f-06d595294e63"). InnerVolumeSpecName "kube-api-access-lg62x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:49.413074 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:49.413044 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg62x\" (UniqueName: \"kubernetes.io/projected/f8f10885-315c-4312-a86f-06d595294e63-kube-api-access-lg62x\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:02:49.607395 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:49.607362 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e3b9f4-d540-4717-82b4-cd874ce4db15" path="/var/lib/kubelet/pods/29e3b9f4-d540-4717-82b4-cd874ce4db15/volumes" Oct 30 06:02:49.807498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:49.807429 6999 generic.go:356] "Generic (PLEG): container finished" podID="f8f10885-315c-4312-a86f-06d595294e63" containerID="87b4b53a07c2d24ed7aaa44f63abe62fc0e745c3052dab0f4ea2ca2984dc60c4" exitCode=0 Oct 30 06:02:49.807498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:49.807475 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-bqddx" Oct 30 06:02:49.807700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:49.807519 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bqddx" event={"ID":"f8f10885-315c-4312-a86f-06d595294e63","Type":"ContainerDied","Data":"87b4b53a07c2d24ed7aaa44f63abe62fc0e745c3052dab0f4ea2ca2984dc60c4"} Oct 30 06:02:49.807700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:49.807564 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-bqddx" event={"ID":"f8f10885-315c-4312-a86f-06d595294e63","Type":"ContainerDied","Data":"4b38e180738cf5ca0f644eef01ee83ae905a73f0690becd8631c2d5faa6fba0b"} Oct 30 06:02:49.807700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:49.807587 6999 scope.go:122] "RemoveContainer" containerID="87b4b53a07c2d24ed7aaa44f63abe62fc0e745c3052dab0f4ea2ca2984dc60c4" Oct 30 06:02:49.815956 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:49.815942 6999 scope.go:122] "RemoveContainer" containerID="87b4b53a07c2d24ed7aaa44f63abe62fc0e745c3052dab0f4ea2ca2984dc60c4" Oct 30 06:02:49.816231 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:02:49.816202 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b4b53a07c2d24ed7aaa44f63abe62fc0e745c3052dab0f4ea2ca2984dc60c4\": container with ID starting with 87b4b53a07c2d24ed7aaa44f63abe62fc0e745c3052dab0f4ea2ca2984dc60c4 not found: ID does not exist" containerID="87b4b53a07c2d24ed7aaa44f63abe62fc0e745c3052dab0f4ea2ca2984dc60c4" Oct 30 06:02:49.816278 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:49.816239 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b4b53a07c2d24ed7aaa44f63abe62fc0e745c3052dab0f4ea2ca2984dc60c4"} err="failed to get container status \"87b4b53a07c2d24ed7aaa44f63abe62fc0e745c3052dab0f4ea2ca2984dc60c4\": rpc error: code = NotFound desc = could not find container \"87b4b53a07c2d24ed7aaa44f63abe62fc0e745c3052dab0f4ea2ca2984dc60c4\": container with ID starting with 87b4b53a07c2d24ed7aaa44f63abe62fc0e745c3052dab0f4ea2ca2984dc60c4 not found: ID does not exist" Oct 30 06:02:49.856867 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:49.856848 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bqddx"] Oct 30 06:02:49.860710 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:49.860693 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-bqddx"] Oct 30 06:02:51.614728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:51.614685 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f10885-315c-4312-a86f-06d595294e63" path="/var/lib/kubelet/pods/f8f10885-315c-4312-a86f-06d595294e63/volumes" Oct 30 06:02:55.691470 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:55.691444 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" Oct 30 06:02:56.697810 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:56.697780 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-dd578" Oct 30 06:02:57.209926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:02:57.209898 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:03:04.141487 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:03:04.141398 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:03:14.314290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:14.314255 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:03:15.158522 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:15.158485 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:03:30.307556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:30.307526 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5xns7"] Oct 30 06:03:30.308161 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:30.307920 6999 cpu_manager.go:414] "containerMap: removing container" podUID="f8f10885-315c-4312-a86f-06d595294e63" containerName="authorino" Oct 30 06:03:30.308161 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:30.307934 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f10885-315c-4312-a86f-06d595294e63" containerName="authorino" Oct 30 06:03:30.308161 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:30.308029 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f10885-315c-4312-a86f-06d595294e63" containerName="authorino" Oct 30 06:03:30.311405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:30.311387 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xns7" Oct 30 06:03:30.332883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:30.332860 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5xns7"] Oct 30 06:03:30.365844 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:30.365816 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2c64f13-36fe-4739-930b-f094dd0e225c-cert\") pod \"llmisvc-controller-manager-6466c554d8-5xns7\" (UID: \"c2c64f13-36fe-4739-930b-f094dd0e225c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xns7" Oct 30 06:03:30.365945 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:30.365868 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm7rt\" (UniqueName: \"kubernetes.io/projected/c2c64f13-36fe-4739-930b-f094dd0e225c-kube-api-access-nm7rt\") pod \"llmisvc-controller-manager-6466c554d8-5xns7\" (UID: \"c2c64f13-36fe-4739-930b-f094dd0e225c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xns7" Oct 30 06:03:30.467133 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:30.467107 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2c64f13-36fe-4739-930b-f094dd0e225c-cert\") pod \"llmisvc-controller-manager-6466c554d8-5xns7\" (UID: \"c2c64f13-36fe-4739-930b-f094dd0e225c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xns7" Oct 30 06:03:30.467301 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:30.467153 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm7rt\" (UniqueName: \"kubernetes.io/projected/c2c64f13-36fe-4739-930b-f094dd0e225c-kube-api-access-nm7rt\") pod \"llmisvc-controller-manager-6466c554d8-5xns7\" (UID: \"c2c64f13-36fe-4739-930b-f094dd0e225c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xns7" Oct 30 06:03:30.469698 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:30.469676 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2c64f13-36fe-4739-930b-f094dd0e225c-cert\") pod \"llmisvc-controller-manager-6466c554d8-5xns7\" (UID: \"c2c64f13-36fe-4739-930b-f094dd0e225c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xns7" Oct 30 06:03:30.492549 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:30.492521 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm7rt\" (UniqueName: \"kubernetes.io/projected/c2c64f13-36fe-4739-930b-f094dd0e225c-kube-api-access-nm7rt\") pod \"llmisvc-controller-manager-6466c554d8-5xns7\" (UID: \"c2c64f13-36fe-4739-930b-f094dd0e225c\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xns7" Oct 30 06:03:30.624179 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:30.624157 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xns7" Oct 30 06:03:30.921785 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:30.921760 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5xns7"] Oct 30 06:03:30.922713 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:03:30.922688 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c64f13_36fe_4739_930b_f094dd0e225c.slice/crio-898b0c4b37d9b8db70c61b763d07dd1ccfe10e729dd0df33384c65c43e7949d6 WatchSource:0}: Error finding container 898b0c4b37d9b8db70c61b763d07dd1ccfe10e729dd0df33384c65c43e7949d6: Status 404 returned error can't find the container with id 898b0c4b37d9b8db70c61b763d07dd1ccfe10e729dd0df33384c65c43e7949d6 Oct 30 06:03:30.994447 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:30.994419 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xns7" event={"ID":"c2c64f13-36fe-4739-930b-f094dd0e225c","Type":"ContainerStarted","Data":"898b0c4b37d9b8db70c61b763d07dd1ccfe10e729dd0df33384c65c43e7949d6"} Oct 30 06:03:32.000745 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:32.000715 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xns7" event={"ID":"c2c64f13-36fe-4739-930b-f094dd0e225c","Type":"ContainerStarted","Data":"64250c9e7f1cae68c6cb3559eeb159caac3a13a1a7a956a33cc0f323a16f662d"} Oct 30 06:03:32.001130 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:32.000854 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xns7" Oct 30 06:03:32.041621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:32.041572 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xns7" podStartSLOduration=1.712297821 podStartE2EDuration="2.041545543s" totalImagesPullingTime="329.247722ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:03:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:32.0407701 +0000 UTC m=+578.549789996" watchObservedRunningTime="2026-10-30 06:03:32.041545543 +0000 UTC m=+578.550565427" Oct 30 06:03:34.140995 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:03:34.140952 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:03:57.342425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:03:57.342394 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:04:03.007675 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:03.007648 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xns7" Oct 30 06:04:03.082895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:03.082856 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw"] Oct 30 06:04:03.083180 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:03.083151 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" podUID="683de426-d0ef-4eb9-8c8f-3ca3e2a066a4" containerName="manager" containerID="cri-o://0aa7ec7d9c4c6410b5d090b1cf089d03310c5e031aee2f6194743f2cf5038162" gracePeriod=30 Oct 30 06:04:03.335187 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:03.335163 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" Oct 30 06:04:03.449080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:03.449048 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/683de426-d0ef-4eb9-8c8f-3ca3e2a066a4-cert\" (UniqueName: \"kubernetes.io/secret/683de426-d0ef-4eb9-8c8f-3ca3e2a066a4-cert\") pod \"683de426-d0ef-4eb9-8c8f-3ca3e2a066a4\" (UID: \"683de426-d0ef-4eb9-8c8f-3ca3e2a066a4\") " Oct 30 06:04:03.449274 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:03.449106 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/683de426-d0ef-4eb9-8c8f-3ca3e2a066a4-kube-api-access-72rg7\" (UniqueName: \"kubernetes.io/projected/683de426-d0ef-4eb9-8c8f-3ca3e2a066a4-kube-api-access-72rg7\") pod \"683de426-d0ef-4eb9-8c8f-3ca3e2a066a4\" (UID: \"683de426-d0ef-4eb9-8c8f-3ca3e2a066a4\") " Oct 30 06:04:03.451445 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:03.451417 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/683de426-d0ef-4eb9-8c8f-3ca3e2a066a4-cert" pod "683de426-d0ef-4eb9-8c8f-3ca3e2a066a4" (UID: "683de426-d0ef-4eb9-8c8f-3ca3e2a066a4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:04:03.451556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:03.451442 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/683de426-d0ef-4eb9-8c8f-3ca3e2a066a4-kube-api-access-72rg7" pod "683de426-d0ef-4eb9-8c8f-3ca3e2a066a4" (UID: "683de426-d0ef-4eb9-8c8f-3ca3e2a066a4"). InnerVolumeSpecName "kube-api-access-72rg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:04:03.550599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:03.550575 6999 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/683de426-d0ef-4eb9-8c8f-3ca3e2a066a4-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:04:03.550599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:03.550599 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72rg7\" (UniqueName: \"kubernetes.io/projected/683de426-d0ef-4eb9-8c8f-3ca3e2a066a4-kube-api-access-72rg7\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:04:04.141051 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:04:04.141017 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:04:04.146433 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:04.146410 6999 generic.go:356] "Generic (PLEG): container finished" podID="683de426-d0ef-4eb9-8c8f-3ca3e2a066a4" containerID="0aa7ec7d9c4c6410b5d090b1cf089d03310c5e031aee2f6194743f2cf5038162" exitCode=0 Oct 30 06:04:04.146561 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:04.146453 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" Oct 30 06:04:04.146561 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:04.146487 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" event={"ID":"683de426-d0ef-4eb9-8c8f-3ca3e2a066a4","Type":"ContainerDied","Data":"0aa7ec7d9c4c6410b5d090b1cf089d03310c5e031aee2f6194743f2cf5038162"} Oct 30 06:04:04.146561 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:04.146545 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw" event={"ID":"683de426-d0ef-4eb9-8c8f-3ca3e2a066a4","Type":"ContainerDied","Data":"4f5404d48084c43b87b322330db9837806775df6600aec37297e1d34d2aff55c"} Oct 30 06:04:04.146663 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:04.146571 6999 scope.go:122] "RemoveContainer" containerID="0aa7ec7d9c4c6410b5d090b1cf089d03310c5e031aee2f6194743f2cf5038162" Oct 30 06:04:04.155019 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:04.155002 6999 scope.go:122] "RemoveContainer" containerID="0aa7ec7d9c4c6410b5d090b1cf089d03310c5e031aee2f6194743f2cf5038162" Oct 30 06:04:04.155317 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:04:04.155297 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa7ec7d9c4c6410b5d090b1cf089d03310c5e031aee2f6194743f2cf5038162\": container with ID starting with 0aa7ec7d9c4c6410b5d090b1cf089d03310c5e031aee2f6194743f2cf5038162 not found: ID does not exist" containerID="0aa7ec7d9c4c6410b5d090b1cf089d03310c5e031aee2f6194743f2cf5038162" Oct 30 06:04:04.155389 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:04.155325 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa7ec7d9c4c6410b5d090b1cf089d03310c5e031aee2f6194743f2cf5038162"} err="failed to get container status \"0aa7ec7d9c4c6410b5d090b1cf089d03310c5e031aee2f6194743f2cf5038162\": rpc error: code = NotFound desc = could not find container \"0aa7ec7d9c4c6410b5d090b1cf089d03310c5e031aee2f6194743f2cf5038162\": container with ID starting with 0aa7ec7d9c4c6410b5d090b1cf089d03310c5e031aee2f6194743f2cf5038162 not found: ID does not exist" Oct 30 06:04:04.200279 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:04.200256 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw"] Oct 30 06:04:04.204157 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:04.204136 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2lbgw"] Oct 30 06:04:04.274441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:04.274414 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:04:04.534792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:04.534719 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:04:05.607422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:05.607390 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="683de426-d0ef-4eb9-8c8f-3ca3e2a066a4" path="/var/lib/kubelet/pods/683de426-d0ef-4eb9-8c8f-3ca3e2a066a4/volumes" Oct 30 06:04:09.488478 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:09.488446 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:04:09.715648 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:09.715613 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:04:33.285920 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:33.285887 6999 generic.go:356] "Generic (PLEG): container finished" podID="9c87696f-3650-470f-9ddf-2d1137017d59" containerID="209d4a7a75ef9f03477b20553285fc731e2a519cea74bb4b5d9987ad20f2babb" exitCode=1 Oct 30 06:04:33.286354 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:33.285962 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5749bffc96-tr2hn" event={"ID":"9c87696f-3650-470f-9ddf-2d1137017d59","Type":"ContainerDied","Data":"209d4a7a75ef9f03477b20553285fc731e2a519cea74bb4b5d9987ad20f2babb"} Oct 30 06:04:33.286354 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:33.286291 6999 scope.go:122] "RemoveContainer" containerID="209d4a7a75ef9f03477b20553285fc731e2a519cea74bb4b5d9987ad20f2babb" Oct 30 06:04:33.532023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:33.531949 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:04:34.044949 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:34.044883 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:04:34.140908 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:04:34.140869 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:04:34.292476 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:34.292449 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5749bffc96-tr2hn" event={"ID":"9c87696f-3650-470f-9ddf-2d1137017d59","Type":"ContainerStarted","Data":"48317b3fd2ecb0d2e7b9f28d089f3781a9de6bbe085398a7b266b952c7b6633b"} Oct 30 06:04:34.292839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:34.292651 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5749bffc96-tr2hn" Oct 30 06:04:35.362812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:35.362778 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:04:35.528597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:35.528565 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:04:45.298066 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:45.298038 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5749bffc96-tr2hn" Oct 30 06:04:51.105652 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:51.105618 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:04:57.442430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:57.442394 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:04:58.231505 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:04:58.231468 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:05:00.587530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:00.587484 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:05:02.577788 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:02.577760 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:05:04.141156 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:05:04.141118 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:05:05.560874 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:05.560844 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:05:07.865738 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:07.865669 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:05:09.774919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.774886 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:05:09.816313 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.816275 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9"] Oct 30 06:05:09.816763 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.816746 6999 cpu_manager.go:414] "containerMap: removing container" podUID="683de426-d0ef-4eb9-8c8f-3ca3e2a066a4" containerName="manager" Oct 30 06:05:09.816763 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.816763 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="683de426-d0ef-4eb9-8c8f-3ca3e2a066a4" containerName="manager" Oct 30 06:05:09.816889 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.816847 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="683de426-d0ef-4eb9-8c8f-3ca3e2a066a4" containerName="manager" Oct 30 06:05:09.819620 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.819605 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:09.851771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.851721 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9"] Oct 30 06:05:09.915513 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.915485 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hc86w\"" Oct 30 06:05:09.932639 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.932610 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 06:05:09.936073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.936046 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:09.936202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.936080 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:09.936202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.936101 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:09.936310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.936228 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:09.936310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.936260 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27965d4b-4c9a-4c68-835f-8f005ff9230d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:09.936310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.936289 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:09.936310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.936305 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78xxc\" (UniqueName: \"kubernetes.io/projected/27965d4b-4c9a-4c68-835f-8f005ff9230d-kube-api-access-78xxc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:09.936452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.936435 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:09.936491 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:09.936470 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.037681 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.037583 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.037681 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.037635 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.037681 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.037673 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.037974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.037696 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.037974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.037726 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.037974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.037785 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.037974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.037818 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27965d4b-4c9a-4c68-835f-8f005ff9230d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.037974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.037854 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.037974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.037876 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78xxc\" (UniqueName: \"kubernetes.io/projected/27965d4b-4c9a-4c68-835f-8f005ff9230d-kube-api-access-78xxc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.038267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.038057 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.038267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.038088 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.038267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.038167 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.038369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.038356 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.038539 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.038519 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27965d4b-4c9a-4c68-835f-8f005ff9230d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.040305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.040276 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.040527 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.040506 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.061453 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.061421 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.061453 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.061448 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78xxc\" (UniqueName: \"kubernetes.io/projected/27965d4b-4c9a-4c68-835f-8f005ff9230d-kube-api-access-78xxc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-q47h9\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.133719 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.133681 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:10.454506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:10.454470 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9"] Oct 30 06:05:10.457883 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:05:10.457853 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27965d4b_4c9a_4c68_835f_8f005ff9230d.slice/crio-f89601088e05b90a2c296fa1b40e574f912ee4cb87cd1ce22602c71bfba80a1f WatchSource:0}: Error finding container f89601088e05b90a2c296fa1b40e574f912ee4cb87cd1ce22602c71bfba80a1f: Status 404 returned error can't find the container with id f89601088e05b90a2c296fa1b40e574f912ee4cb87cd1ce22602c71bfba80a1f Oct 30 06:05:11.479244 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:11.479172 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" event={"ID":"27965d4b-4c9a-4c68-835f-8f005ff9230d","Type":"ContainerStarted","Data":"f89601088e05b90a2c296fa1b40e574f912ee4cb87cd1ce22602c71bfba80a1f"} Oct 30 06:05:11.605904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:11.605850 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:05:13.223068 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:13.223029 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:13.223416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:13.223118 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:13.223416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:13.223162 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:13.491781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:13.491587 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" event={"ID":"27965d4b-4c9a-4c68-835f-8f005ff9230d","Type":"ContainerStarted","Data":"7bf4cd34fb2d7a4baf4145f58948725cba3525f832d93b90d532cd266db831e8"} Oct 30 06:05:13.551352 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:13.551300 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" podStartSLOduration=1.789355031 podStartE2EDuration="4.551284912s" totalImagesPullingTime="2.761929881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:13.549758635 +0000 UTC m=+680.058778522" watchObservedRunningTime="2026-10-30 06:05:13.551284912 +0000 UTC m=+680.060304864" Oct 30 06:05:14.133892 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:14.133852 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:14.138943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:14.138905 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q47h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 06:05:14.139262 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:14.139226 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" podUID="27965d4b-4c9a-4c68-835f-8f005ff9230d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 06:05:14.140268 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:14.140075 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:05:15.141240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:15.140344 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q47h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 06:05:15.141240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:15.140435 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" podUID="27965d4b-4c9a-4c68-835f-8f005ff9230d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 06:05:16.138238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:16.134361 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q47h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 06:05:16.138238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:16.134424 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" podUID="27965d4b-4c9a-4c68-835f-8f005ff9230d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 06:05:17.134112 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:17.134076 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q47h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 06:05:17.134637 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:17.134133 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" podUID="27965d4b-4c9a-4c68-835f-8f005ff9230d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 06:05:18.152240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:18.134200 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q47h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 06:05:18.152890 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:18.152851 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" podUID="27965d4b-4c9a-4c68-835f-8f005ff9230d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 06:05:19.137249 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:19.134198 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q47h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 06:05:19.137249 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:19.134298 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" podUID="27965d4b-4c9a-4c68-835f-8f005ff9230d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 06:05:20.134949 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:20.134914 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q47h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 06:05:20.135402 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:20.134969 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" podUID="27965d4b-4c9a-4c68-835f-8f005ff9230d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 06:05:22.175429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:22.175385 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-q47h9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:05:22.175978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:22.175462 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" podUID="27965d4b-4c9a-4c68-835f-8f005ff9230d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:22.983231 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:22.983170 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:22.983888 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:22.983825 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:22.984714 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:22.984697 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:05:24.888859 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:24.888822 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8"] Oct 30 06:05:24.893037 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:24.893017 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:24.918144 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:24.918100 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8"] Oct 30 06:05:24.987484 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:24.987441 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r48vx\" (UniqueName: \"kubernetes.io/projected/edf39134-4250-4b1c-af52-f5d4f8ddcd10-kube-api-access-r48vx\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:24.987656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:24.987546 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf39134-4250-4b1c-af52-f5d4f8ddcd10-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:24.987656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:24.987589 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf39134-4250-4b1c-af52-f5d4f8ddcd10-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:24.987732 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:24.987652 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf39134-4250-4b1c-af52-f5d4f8ddcd10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:24.987732 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:24.987720 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf39134-4250-4b1c-af52-f5d4f8ddcd10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:24.987795 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:24.987757 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf39134-4250-4b1c-af52-f5d4f8ddcd10-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:24.987828 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:24.987812 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf39134-4250-4b1c-af52-f5d4f8ddcd10-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:24.987887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:24.987868 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf39134-4250-4b1c-af52-f5d4f8ddcd10-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:24.987947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:24.987932 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf39134-4250-4b1c-af52-f5d4f8ddcd10-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.089439 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.089402 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf39134-4250-4b1c-af52-f5d4f8ddcd10-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.089647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.089450 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf39134-4250-4b1c-af52-f5d4f8ddcd10-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.089647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.089473 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf39134-4250-4b1c-af52-f5d4f8ddcd10-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.089647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.089495 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf39134-4250-4b1c-af52-f5d4f8ddcd10-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.089647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.089551 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r48vx\" (UniqueName: \"kubernetes.io/projected/edf39134-4250-4b1c-af52-f5d4f8ddcd10-kube-api-access-r48vx\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.089647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.089584 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf39134-4250-4b1c-af52-f5d4f8ddcd10-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.089925 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.089904 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf39134-4250-4b1c-af52-f5d4f8ddcd10-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.089990 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.089938 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/edf39134-4250-4b1c-af52-f5d4f8ddcd10-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.090053 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.089987 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/edf39134-4250-4b1c-af52-f5d4f8ddcd10-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.090053 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.090010 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf39134-4250-4b1c-af52-f5d4f8ddcd10-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.090159 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.090096 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf39134-4250-4b1c-af52-f5d4f8ddcd10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.090159 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.090098 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/edf39134-4250-4b1c-af52-f5d4f8ddcd10-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.090285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.090157 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf39134-4250-4b1c-af52-f5d4f8ddcd10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.090788 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.090763 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/edf39134-4250-4b1c-af52-f5d4f8ddcd10-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.092340 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.092318 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/edf39134-4250-4b1c-af52-f5d4f8ddcd10-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.092891 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.092876 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/edf39134-4250-4b1c-af52-f5d4f8ddcd10-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.112871 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.112845 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r48vx\" (UniqueName: \"kubernetes.io/projected/edf39134-4250-4b1c-af52-f5d4f8ddcd10-kube-api-access-r48vx\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.112980 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.112965 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/edf39134-4250-4b1c-af52-f5d4f8ddcd10-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mlqh8\" (UID: \"edf39134-4250-4b1c-af52-f5d4f8ddcd10\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.210344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.210266 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:25.534034 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.533961 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8"] Oct 30 06:05:25.537302 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:05:25.537274 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf39134_4250_4b1c_af52_f5d4f8ddcd10.slice/crio-cd23ea84d8f676f4e586c031dc9eea054373b15e85a503dd8addbafb641ca2db WatchSource:0}: Error finding container cd23ea84d8f676f4e586c031dc9eea054373b15e85a503dd8addbafb641ca2db: Status 404 returned error can't find the container with id cd23ea84d8f676f4e586c031dc9eea054373b15e85a503dd8addbafb641ca2db Oct 30 06:05:25.539519 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.539488 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:25.539595 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.539555 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:25.539595 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.539582 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:25.553551 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:25.553527 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" event={"ID":"edf39134-4250-4b1c-af52-f5d4f8ddcd10","Type":"ContainerStarted","Data":"cd23ea84d8f676f4e586c031dc9eea054373b15e85a503dd8addbafb641ca2db"} Oct 30 06:05:26.565257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:26.565198 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" event={"ID":"edf39134-4250-4b1c-af52-f5d4f8ddcd10","Type":"ContainerStarted","Data":"d08fcbe501a1837b893b7cf249f0fe3952862f6c7f2a76d34bb801ac55457c21"} Oct 30 06:05:27.210985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:27.210948 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:27.212333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:27.212309 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mlqh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 06:05:27.212410 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:27.212371 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" podUID="edf39134-4250-4b1c-af52-f5d4f8ddcd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 06:05:28.210933 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:28.210896 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mlqh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 06:05:28.211349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:28.210967 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" podUID="edf39134-4250-4b1c-af52-f5d4f8ddcd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 06:05:29.210927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:29.210885 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mlqh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 06:05:29.211108 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:29.210948 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" podUID="edf39134-4250-4b1c-af52-f5d4f8ddcd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 06:05:30.211257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:30.211192 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mlqh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 06:05:30.211660 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:30.211283 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" podUID="edf39134-4250-4b1c-af52-f5d4f8ddcd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 06:05:31.210793 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:31.210759 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mlqh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 06:05:31.210979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:31.210812 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" podUID="edf39134-4250-4b1c-af52-f5d4f8ddcd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 06:05:32.210692 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:32.210646 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mlqh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 06:05:32.211094 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:32.210723 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" podUID="edf39134-4250-4b1c-af52-f5d4f8ddcd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 06:05:33.825509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:33.825444 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" podStartSLOduration=9.82542525 podStartE2EDuration="9.82542525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:26.633672943 +0000 UTC m=+693.142692830" watchObservedRunningTime="2026-10-30 06:05:33.82542525 +0000 UTC m=+700.334445138" Oct 30 06:05:33.843894 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:33.843855 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674cd869b7-wvxv5"] Oct 30 06:05:33.844351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:33.844323 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-674cd869b7-wvxv5" podUID="8d90f37f-b29a-4577-ae53-7e8ca34bda9a" containerName="authorino" containerID="cri-o://289d0448449c9a055b6c8d906d51cec5bc5aee92b9f9f53161b35d53bff262c4" gracePeriod=30 Oct 30 06:05:33.846276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:33.846258 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:33.847828 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:33.847806 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:33.859592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:33.859565 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mlqh8" Oct 30 06:05:34.108632 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.108604 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674cd869b7-wvxv5" Oct 30 06:05:34.142054 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:05:34.142013 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:05:34.202944 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.202912 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d90f37f-b29a-4577-ae53-7e8ca34bda9a-kube-api-access-85v28\" (UniqueName: \"kubernetes.io/projected/8d90f37f-b29a-4577-ae53-7e8ca34bda9a-kube-api-access-85v28\") pod \"8d90f37f-b29a-4577-ae53-7e8ca34bda9a\" (UID: \"8d90f37f-b29a-4577-ae53-7e8ca34bda9a\") " Oct 30 06:05:34.203125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.203032 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d90f37f-b29a-4577-ae53-7e8ca34bda9a-tls-cert\" (UniqueName: \"kubernetes.io/secret/8d90f37f-b29a-4577-ae53-7e8ca34bda9a-tls-cert\") pod \"8d90f37f-b29a-4577-ae53-7e8ca34bda9a\" (UID: \"8d90f37f-b29a-4577-ae53-7e8ca34bda9a\") " Oct 30 06:05:34.205250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.205206 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d90f37f-b29a-4577-ae53-7e8ca34bda9a-kube-api-access-85v28" pod "8d90f37f-b29a-4577-ae53-7e8ca34bda9a" (UID: "8d90f37f-b29a-4577-ae53-7e8ca34bda9a"). InnerVolumeSpecName "kube-api-access-85v28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:34.214336 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.214313 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d90f37f-b29a-4577-ae53-7e8ca34bda9a-tls-cert" pod "8d90f37f-b29a-4577-ae53-7e8ca34bda9a" (UID: "8d90f37f-b29a-4577-ae53-7e8ca34bda9a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:05:34.304700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.304670 6999 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8d90f37f-b29a-4577-ae53-7e8ca34bda9a-tls-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:05:34.304871 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.304704 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85v28\" (UniqueName: \"kubernetes.io/projected/8d90f37f-b29a-4577-ae53-7e8ca34bda9a-kube-api-access-85v28\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:05:34.608091 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.608054 6999 generic.go:356] "Generic (PLEG): container finished" podID="8d90f37f-b29a-4577-ae53-7e8ca34bda9a" containerID="289d0448449c9a055b6c8d906d51cec5bc5aee92b9f9f53161b35d53bff262c4" exitCode=0 Oct 30 06:05:34.608288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.608097 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674cd869b7-wvxv5" Oct 30 06:05:34.608288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.608140 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674cd869b7-wvxv5" event={"ID":"8d90f37f-b29a-4577-ae53-7e8ca34bda9a","Type":"ContainerDied","Data":"289d0448449c9a055b6c8d906d51cec5bc5aee92b9f9f53161b35d53bff262c4"} Oct 30 06:05:34.608288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.608181 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674cd869b7-wvxv5" event={"ID":"8d90f37f-b29a-4577-ae53-7e8ca34bda9a","Type":"ContainerDied","Data":"586a6b5edcdcbd43fa7e59cb7b7042c8f29cc10bce0d231db9853b241ed688e9"} Oct 30 06:05:34.608288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.608198 6999 scope.go:122] "RemoveContainer" containerID="289d0448449c9a055b6c8d906d51cec5bc5aee92b9f9f53161b35d53bff262c4" Oct 30 06:05:34.617105 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.617088 6999 scope.go:122] "RemoveContainer" containerID="289d0448449c9a055b6c8d906d51cec5bc5aee92b9f9f53161b35d53bff262c4" Oct 30 06:05:34.617352 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:05:34.617334 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"289d0448449c9a055b6c8d906d51cec5bc5aee92b9f9f53161b35d53bff262c4\": container with ID starting with 289d0448449c9a055b6c8d906d51cec5bc5aee92b9f9f53161b35d53bff262c4 not found: ID does not exist" containerID="289d0448449c9a055b6c8d906d51cec5bc5aee92b9f9f53161b35d53bff262c4" Oct 30 06:05:34.617405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.617362 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"289d0448449c9a055b6c8d906d51cec5bc5aee92b9f9f53161b35d53bff262c4"} err="failed to get container status \"289d0448449c9a055b6c8d906d51cec5bc5aee92b9f9f53161b35d53bff262c4\": rpc error: code = NotFound desc = could not find container \"289d0448449c9a055b6c8d906d51cec5bc5aee92b9f9f53161b35d53bff262c4\": container with ID starting with 289d0448449c9a055b6c8d906d51cec5bc5aee92b9f9f53161b35d53bff262c4 not found: ID does not exist" Oct 30 06:05:34.679274 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.679237 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674cd869b7-wvxv5"] Oct 30 06:05:34.686622 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:34.686579 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-674cd869b7-wvxv5"] Oct 30 06:05:35.164808 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:35.164781 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:05:35.334780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:35.334751 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:05:35.608394 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:35.608370 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d90f37f-b29a-4577-ae53-7e8ca34bda9a" path="/var/lib/kubelet/pods/8d90f37f-b29a-4577-ae53-7e8ca34bda9a/volumes" Oct 30 06:05:36.854277 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:36.854234 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:05:37.066576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:37.066542 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:05:39.915182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:39.915148 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w"] Oct 30 06:05:39.915707 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:39.915575 6999 cpu_manager.go:414] "containerMap: removing container" podUID="8d90f37f-b29a-4577-ae53-7e8ca34bda9a" containerName="authorino" Oct 30 06:05:39.915707 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:39.915586 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d90f37f-b29a-4577-ae53-7e8ca34bda9a" containerName="authorino" Oct 30 06:05:39.915707 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:39.915678 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d90f37f-b29a-4577-ae53-7e8ca34bda9a" containerName="authorino" Oct 30 06:05:39.918944 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:39.918924 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:39.948566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:39.948531 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w"] Oct 30 06:05:40.061875 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.061839 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.062071 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.061892 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.062071 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.061983 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.062071 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.062037 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.062278 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.062089 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.062278 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.062134 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.062278 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.062196 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ztp4\" (UniqueName: \"kubernetes.io/projected/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-kube-api-access-6ztp4\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.062429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.062267 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.062429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.062327 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.163303 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.163268 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.163303 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.163303 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.163564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.163343 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ztp4\" (UniqueName: \"kubernetes.io/projected/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-kube-api-access-6ztp4\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.163564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.163364 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.163564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.163379 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.163564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.163421 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.163564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.163444 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.163564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.163487 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.163564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.163554 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.163931 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.163880 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.163931 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.163911 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.164048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.163988 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.164088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.164046 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.164570 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.164544 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.165885 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.165835 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.166060 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.166043 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.196989 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.196959 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.197272 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.197247 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ztp4\" (UniqueName: \"kubernetes.io/projected/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-kube-api-access-6ztp4\") pod \"maas-default-gateway-openshift-default-58847f9fc5-54g5w\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.232115 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.232088 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:40.554535 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.554458 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w"] Oct 30 06:05:40.557456 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:05:40.557431 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7c4be1_b8f3_4c44_81f5_4bd91655897f.slice/crio-d46aaf81e147cf71a2e3a964b0f25c92627da1290266ecfda2d65d8a41a068d4 WatchSource:0}: Error finding container d46aaf81e147cf71a2e3a964b0f25c92627da1290266ecfda2d65d8a41a068d4: Status 404 returned error can't find the container with id d46aaf81e147cf71a2e3a964b0f25c92627da1290266ecfda2d65d8a41a068d4 Oct 30 06:05:40.559856 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.559824 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:40.559917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.559895 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:40.559956 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.559930 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:40.639908 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.639873 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" event={"ID":"9a7c4be1-b8f3-4c44-81f5-4bd91655897f","Type":"ContainerStarted","Data":"e02d911e83e6d7c35bba97cc068f620c2936a3af8cdc77520e6b235bb1ef4141"} Oct 30 06:05:40.640059 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:40.639933 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" event={"ID":"9a7c4be1-b8f3-4c44-81f5-4bd91655897f","Type":"ContainerStarted","Data":"d46aaf81e147cf71a2e3a964b0f25c92627da1290266ecfda2d65d8a41a068d4"} Oct 30 06:05:41.232955 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:41.232920 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:41.244239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:41.234256 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-54g5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 06:05:41.244239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:41.234316 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" podUID="9a7c4be1-b8f3-4c44-81f5-4bd91655897f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 06:05:42.233238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:42.233174 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-54g5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 06:05:42.233651 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:42.233254 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" podUID="9a7c4be1-b8f3-4c44-81f5-4bd91655897f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 06:05:43.233122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:43.233082 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-54g5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 06:05:43.233332 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:43.233143 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" podUID="9a7c4be1-b8f3-4c44-81f5-4bd91655897f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 06:05:44.233457 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:44.233417 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-54g5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 06:05:44.233843 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:44.233476 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" podUID="9a7c4be1-b8f3-4c44-81f5-4bd91655897f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 06:05:45.232886 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:45.232844 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-54g5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 06:05:45.233077 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:45.232911 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" podUID="9a7c4be1-b8f3-4c44-81f5-4bd91655897f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 06:05:46.232776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:46.232719 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-54g5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 06:05:46.233176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:46.232804 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" podUID="9a7c4be1-b8f3-4c44-81f5-4bd91655897f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 06:05:47.233135 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:47.233092 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-54g5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 06:05:47.233585 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:47.233179 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" podUID="9a7c4be1-b8f3-4c44-81f5-4bd91655897f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 06:05:48.950752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:48.950718 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:48.951429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:48.951405 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:48.952276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:48.952252 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:05:49.007897 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:49.007847 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" podStartSLOduration=10.007832578 podStartE2EDuration="10.007832578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:40.694324768 +0000 UTC m=+707.203344656" watchObservedRunningTime="2026-10-30 06:05:49.007832578 +0000 UTC m=+715.516852465" Oct 30 06:05:57.579559 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:57.579520 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:05:59.662871 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:05:59.662839 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:04.141271 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:06:04.141174 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:06:06.207941 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:06.207911 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:06.387160 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:06.387132 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:06.542915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:06.542836 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:06.814017 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:06.813946 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:07.036504 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:07.036471 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:07.250962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:07.250932 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:07.445232 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:07.445179 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:07.683814 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:07.683783 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:08.231051 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:08.231016 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:08.702641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:08.702611 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:08.857881 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:08.857847 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:10.052773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:10.052744 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:10.271352 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:10.271320 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:10.425363 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:10.425334 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:11.091657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:11.091628 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:11.230835 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:11.230806 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:11.697240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:11.697184 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:11.841392 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:11.841358 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:12.097857 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:12.097786 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:12.188265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:12.188202 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx"] Oct 30 06:06:12.192157 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:12.192131 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" Oct 30 06:06:12.223334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:12.223302 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-798193ae-dockercfg-ql5k8\"" Oct 30 06:06:12.224184 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:12.224159 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx"] Oct 30 06:06:12.252563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:12.252538 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-798193ae\"" Oct 30 06:06:12.363614 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:12.363582 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx\" (UID: \"0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8\") " pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" Oct 30 06:06:12.363752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:12.363622 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw5ck\" (UniqueName: \"kubernetes.io/projected/0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8-kube-api-access-fw5ck\") pod \"payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx\" (UID: \"0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8\") " pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" Oct 30 06:06:12.464583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:12.464558 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx\" (UID: \"0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8\") " pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" Oct 30 06:06:12.464726 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:12.464597 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw5ck\" (UniqueName: \"kubernetes.io/projected/0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8-kube-api-access-fw5ck\") pod \"payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx\" (UID: \"0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8\") " pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" Oct 30 06:06:12.465256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:12.465239 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8-plugins-config-volume\") pod \"payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx\" (UID: \"0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8\") " pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" Oct 30 06:06:12.487785 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:12.487757 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw5ck\" (UniqueName: \"kubernetes.io/projected/0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8-kube-api-access-fw5ck\") pod \"payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx\" (UID: \"0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8\") " pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" Oct 30 06:06:12.507655 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:12.507632 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" Oct 30 06:06:12.819892 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:06:12.819864 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1edbe2_0ec3_4ea0_a85b_e77480bc43d8.slice/crio-9285a997d64c542c2adf0d90188e0b7e9e6d7ca6d1aed6aa950294a7d54d2f9c WatchSource:0}: Error finding container 9285a997d64c542c2adf0d90188e0b7e9e6d7ca6d1aed6aa950294a7d54d2f9c: Status 404 returned error can't find the container with id 9285a997d64c542c2adf0d90188e0b7e9e6d7ca6d1aed6aa950294a7d54d2f9c Oct 30 06:06:12.821661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:12.821630 6999 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:06:12.822683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:12.822667 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx"] Oct 30 06:06:13.054709 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:13.054682 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:13.218475 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:13.218445 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:13.807073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:13.807039 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:13.815164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:13.815124 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" event={"ID":"0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8","Type":"ContainerStarted","Data":"9285a997d64c542c2adf0d90188e0b7e9e6d7ca6d1aed6aa950294a7d54d2f9c"} Oct 30 06:06:13.954398 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:13.954357 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:14.037061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.037014 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff"] Oct 30 06:06:14.047252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.047135 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" Oct 30 06:06:14.058910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.058832 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff"] Oct 30 06:06:14.083463 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.083442 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c45a2ff8-dockercfg-55szw\"" Oct 30 06:06:14.083581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.083520 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c45a2ff8\"" Oct 30 06:06:14.184648 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.184614 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2511a9a4-7e0d-4006-a550-2a05b060566f-plugins-config-volume\") pod \"payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff\" (UID: \"2511a9a4-7e0d-4006-a550-2a05b060566f\") " pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" Oct 30 06:06:14.184807 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.184726 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p448d\" (UniqueName: \"kubernetes.io/projected/2511a9a4-7e0d-4006-a550-2a05b060566f-kube-api-access-p448d\") pod \"payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff\" (UID: \"2511a9a4-7e0d-4006-a550-2a05b060566f\") " pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" Oct 30 06:06:14.286109 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.286080 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2511a9a4-7e0d-4006-a550-2a05b060566f-plugins-config-volume\") pod \"payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff\" (UID: \"2511a9a4-7e0d-4006-a550-2a05b060566f\") " pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" Oct 30 06:06:14.286622 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.286143 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p448d\" (UniqueName: \"kubernetes.io/projected/2511a9a4-7e0d-4006-a550-2a05b060566f-kube-api-access-p448d\") pod \"payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff\" (UID: \"2511a9a4-7e0d-4006-a550-2a05b060566f\") " pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" Oct 30 06:06:14.286898 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.286869 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2511a9a4-7e0d-4006-a550-2a05b060566f-plugins-config-volume\") pod \"payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff\" (UID: \"2511a9a4-7e0d-4006-a550-2a05b060566f\") " pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" Oct 30 06:06:14.309825 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.309759 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p448d\" (UniqueName: \"kubernetes.io/projected/2511a9a4-7e0d-4006-a550-2a05b060566f-kube-api-access-p448d\") pod \"payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff\" (UID: \"2511a9a4-7e0d-4006-a550-2a05b060566f\") " pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" Oct 30 06:06:14.386293 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.386253 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" Oct 30 06:06:14.731503 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.730560 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff"] Oct 30 06:06:14.733587 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:06:14.733549 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2511a9a4_7e0d_4006_a550_2a05b060566f.slice/crio-c7fb60b9de6af3f5875093843b544ba6ee74ff62f3596691cefed36448da53f2 WatchSource:0}: Error finding container c7fb60b9de6af3f5875093843b544ba6ee74ff62f3596691cefed36448da53f2: Status 404 returned error can't find the container with id c7fb60b9de6af3f5875093843b544ba6ee74ff62f3596691cefed36448da53f2 Oct 30 06:06:14.822964 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.822929 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" event={"ID":"2511a9a4-7e0d-4006-a550-2a05b060566f","Type":"ContainerStarted","Data":"c7fb60b9de6af3f5875093843b544ba6ee74ff62f3596691cefed36448da53f2"} Oct 30 06:06:14.837088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.837061 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:14.981598 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.981513 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds"] Oct 30 06:06:14.985054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:14.985031 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" Oct 30 06:06:15.010060 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.010023 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9c94fbc2-dockercfg-82hcn\"" Oct 30 06:06:15.010240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.010114 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9c94fbc2\"" Oct 30 06:06:15.012490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.012466 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds"] Oct 30 06:06:15.055788 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.055760 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:15.096011 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.095987 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f105f460-ecf2-4ebd-88b7-e19e979c5cfb-plugins-config-volume\") pod \"payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds\" (UID: \"f105f460-ecf2-4ebd-88b7-e19e979c5cfb\") " pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" Oct 30 06:06:15.096110 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.096053 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxmjw\" (UniqueName: \"kubernetes.io/projected/f105f460-ecf2-4ebd-88b7-e19e979c5cfb-kube-api-access-nxmjw\") pod \"payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds\" (UID: \"f105f460-ecf2-4ebd-88b7-e19e979c5cfb\") " pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" Oct 30 06:06:15.197123 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.197091 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxmjw\" (UniqueName: \"kubernetes.io/projected/f105f460-ecf2-4ebd-88b7-e19e979c5cfb-kube-api-access-nxmjw\") pod \"payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds\" (UID: \"f105f460-ecf2-4ebd-88b7-e19e979c5cfb\") " pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" Oct 30 06:06:15.197321 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.197188 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f105f460-ecf2-4ebd-88b7-e19e979c5cfb-plugins-config-volume\") pod \"payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds\" (UID: \"f105f460-ecf2-4ebd-88b7-e19e979c5cfb\") " pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" Oct 30 06:06:15.197889 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.197868 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f105f460-ecf2-4ebd-88b7-e19e979c5cfb-plugins-config-volume\") pod \"payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds\" (UID: \"f105f460-ecf2-4ebd-88b7-e19e979c5cfb\") " pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" Oct 30 06:06:15.222548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.222524 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxmjw\" (UniqueName: \"kubernetes.io/projected/f105f460-ecf2-4ebd-88b7-e19e979c5cfb-kube-api-access-nxmjw\") pod \"payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds\" (UID: \"f105f460-ecf2-4ebd-88b7-e19e979c5cfb\") " pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" Oct 30 06:06:15.299928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.299903 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" Oct 30 06:06:15.610571 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.610530 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds"] Oct 30 06:06:15.612672 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:06:15.612646 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf105f460_ecf2_4ebd_88b7_e19e979c5cfb.slice/crio-6a3ad3324badb307a3c4bde5b0bd1674100e0c58db9bf7d9e0ade818076ea57e WatchSource:0}: Error finding container 6a3ad3324badb307a3c4bde5b0bd1674100e0c58db9bf7d9e0ade818076ea57e: Status 404 returned error can't find the container with id 6a3ad3324badb307a3c4bde5b0bd1674100e0c58db9bf7d9e0ade818076ea57e Oct 30 06:06:15.830188 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.830138 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" event={"ID":"0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8","Type":"ContainerStarted","Data":"1f4f199827b94c005e5e802d37edd875e7153cf0f25154d6fde569bd55b72d98"} Oct 30 06:06:15.830380 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.830233 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" Oct 30 06:06:15.831901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.831877 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" event={"ID":"2511a9a4-7e0d-4006-a550-2a05b060566f","Type":"ContainerStarted","Data":"4ba8d51e6f48bc7276552c91bb4c1fcb1e278ea61328d2a60bf7795e1dbf3570"} Oct 30 06:06:15.832022 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.831985 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" Oct 30 06:06:15.833335 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.833308 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" event={"ID":"f105f460-ecf2-4ebd-88b7-e19e979c5cfb","Type":"ContainerStarted","Data":"b87d5f1e739dcce537832028ffa81fa24fc2e284df8760bc5abe6bd0e69f3f84"} Oct 30 06:06:15.833445 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.833348 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" event={"ID":"f105f460-ecf2-4ebd-88b7-e19e979c5cfb","Type":"ContainerStarted","Data":"6a3ad3324badb307a3c4bde5b0bd1674100e0c58db9bf7d9e0ade818076ea57e"} Oct 30 06:06:15.833445 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.833410 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" Oct 30 06:06:15.856172 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.856144 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:15.889686 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.889586 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" podStartSLOduration=1.6984570909999999 podStartE2EDuration="3.889571985s" totalImagesPullingTime="2.191114894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:15.888646345 +0000 UTC m=+742.397666232" watchObservedRunningTime="2026-10-30 06:06:15.889571985 +0000 UTC m=+742.398591901" Oct 30 06:06:15.931987 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.931931 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" podStartSLOduration=1.931912756 podStartE2EDuration="1.931912756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:15.930118736 +0000 UTC m=+742.439138646" watchObservedRunningTime="2026-10-30 06:06:15.931912756 +0000 UTC m=+742.440932644" Oct 30 06:06:15.979713 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.979654 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" podStartSLOduration=1.4840162430000001 podStartE2EDuration="1.979639632s" totalImagesPullingTime="495.623389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:15.978828763 +0000 UTC m=+742.487848649" watchObservedRunningTime="2026-10-30 06:06:15.979639632 +0000 UTC m=+742.488659521" Oct 30 06:06:15.998125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:15.998093 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g"] Oct 30 06:06:16.002842 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.002819 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" Oct 30 06:06:16.024053 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.024036 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-55140dc4\"" Oct 30 06:06:16.024241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.024225 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-55140dc4-dockercfg-bn2cl\"" Oct 30 06:06:16.032813 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.032782 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g"] Oct 30 06:06:16.086594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.086570 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:16.105569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.105549 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86283741-0147-4374-b410-f4dadeefc42e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g\" (UID: \"86283741-0147-4374-b410-f4dadeefc42e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" Oct 30 06:06:16.105678 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.105597 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqcl5\" (UniqueName: \"kubernetes.io/projected/86283741-0147-4374-b410-f4dadeefc42e-kube-api-access-dqcl5\") pod \"payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g\" (UID: \"86283741-0147-4374-b410-f4dadeefc42e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" Oct 30 06:06:16.206714 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.206637 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86283741-0147-4374-b410-f4dadeefc42e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g\" (UID: \"86283741-0147-4374-b410-f4dadeefc42e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" Oct 30 06:06:16.206714 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.206707 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqcl5\" (UniqueName: \"kubernetes.io/projected/86283741-0147-4374-b410-f4dadeefc42e-kube-api-access-dqcl5\") pod \"payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g\" (UID: \"86283741-0147-4374-b410-f4dadeefc42e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" Oct 30 06:06:16.207555 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.207534 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86283741-0147-4374-b410-f4dadeefc42e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g\" (UID: \"86283741-0147-4374-b410-f4dadeefc42e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" Oct 30 06:06:16.230674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.230648 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqcl5\" (UniqueName: \"kubernetes.io/projected/86283741-0147-4374-b410-f4dadeefc42e-kube-api-access-dqcl5\") pod \"payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g\" (UID: \"86283741-0147-4374-b410-f4dadeefc42e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" Oct 30 06:06:16.262627 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.262602 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:16.318736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.318697 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" Oct 30 06:06:16.446658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.446626 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:16.619618 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.619588 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:16.640176 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:06:16.640151 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86283741_0147_4374_b410_f4dadeefc42e.slice/crio-fe5c41fa63151fc207ee6b71a3026932a68cff894612e98a970eabe5131f6f6f WatchSource:0}: Error finding container fe5c41fa63151fc207ee6b71a3026932a68cff894612e98a970eabe5131f6f6f: Status 404 returned error can't find the container with id fe5c41fa63151fc207ee6b71a3026932a68cff894612e98a970eabe5131f6f6f Oct 30 06:06:16.641057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.641036 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g"] Oct 30 06:06:16.839595 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.839549 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" event={"ID":"86283741-0147-4374-b410-f4dadeefc42e","Type":"ContainerStarted","Data":"3e7abad5261db035f23949ede9446f99c4a071fedb05487e7f06b764bfc79ce9"} Oct 30 06:06:16.839773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.839622 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" event={"ID":"86283741-0147-4374-b410-f4dadeefc42e","Type":"ContainerStarted","Data":"fe5c41fa63151fc207ee6b71a3026932a68cff894612e98a970eabe5131f6f6f"} Oct 30 06:06:16.839773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.839741 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" Oct 30 06:06:16.882187 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:16.882105 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" podStartSLOduration=1.882092501 podStartE2EDuration="1.882092501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:16.8803423 +0000 UTC m=+743.389362187" watchObservedRunningTime="2026-10-30 06:06:16.882092501 +0000 UTC m=+743.391112387" Oct 30 06:06:17.169704 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:17.169601 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:17.371439 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:17.371408 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:17.542933 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:17.542823 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:18.016481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:18.016442 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:18.366841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:18.366814 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:18.550256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:18.550187 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:18.978746 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:18.978712 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:20.537870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:20.537840 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:20.744754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:20.744718 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:20.907944 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:20.907913 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:21.063963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:21.063936 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:22.665017 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:22.664985 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:22.823691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:22.823659 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:22.905367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:22.905309 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r"] Oct 30 06:06:22.909659 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:22.909620 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" Oct 30 06:06:22.939688 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:22.939654 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-3627d700-dockercfg-vdjpv\"" Oct 30 06:06:22.939854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:22.939710 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-3627d700\"" Oct 30 06:06:22.941293 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:22.941267 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r"] Oct 30 06:06:22.980656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:22.980611 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:23.075677 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:23.075650 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh7qh\" (UniqueName: \"kubernetes.io/projected/9b95207c-4d87-4e10-809a-7c4d4fbefe6e-kube-api-access-qh7qh\") pod \"payload-processing-e2e-derive-3627d700-796cffd94-gbs8r\" (UID: \"9b95207c-4d87-4e10-809a-7c4d4fbefe6e\") " pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" Oct 30 06:06:23.075814 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:23.075689 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b95207c-4d87-4e10-809a-7c4d4fbefe6e-plugins-config-volume\") pod \"payload-processing-e2e-derive-3627d700-796cffd94-gbs8r\" (UID: \"9b95207c-4d87-4e10-809a-7c4d4fbefe6e\") " pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" Oct 30 06:06:23.176580 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:23.176507 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh7qh\" (UniqueName: \"kubernetes.io/projected/9b95207c-4d87-4e10-809a-7c4d4fbefe6e-kube-api-access-qh7qh\") pod \"payload-processing-e2e-derive-3627d700-796cffd94-gbs8r\" (UID: \"9b95207c-4d87-4e10-809a-7c4d4fbefe6e\") " pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" Oct 30 06:06:23.176580 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:23.176546 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b95207c-4d87-4e10-809a-7c4d4fbefe6e-plugins-config-volume\") pod \"payload-processing-e2e-derive-3627d700-796cffd94-gbs8r\" (UID: \"9b95207c-4d87-4e10-809a-7c4d4fbefe6e\") " pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" Oct 30 06:06:23.177320 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:23.177302 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b95207c-4d87-4e10-809a-7c4d4fbefe6e-plugins-config-volume\") pod \"payload-processing-e2e-derive-3627d700-796cffd94-gbs8r\" (UID: \"9b95207c-4d87-4e10-809a-7c4d4fbefe6e\") " pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" Oct 30 06:06:23.202436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:23.202414 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh7qh\" (UniqueName: \"kubernetes.io/projected/9b95207c-4d87-4e10-809a-7c4d4fbefe6e-kube-api-access-qh7qh\") pod \"payload-processing-e2e-derive-3627d700-796cffd94-gbs8r\" (UID: \"9b95207c-4d87-4e10-809a-7c4d4fbefe6e\") " pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" Oct 30 06:06:23.224284 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:23.224262 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" Oct 30 06:06:23.537579 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:23.537552 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r"] Oct 30 06:06:23.538366 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:06:23.538339 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b95207c_4d87_4e10_809a_7c4d4fbefe6e.slice/crio-5c66adee0f128375997a9257aa201e71f43261e938e365962240d9567fe219d6 WatchSource:0}: Error finding container 5c66adee0f128375997a9257aa201e71f43261e938e365962240d9567fe219d6: Status 404 returned error can't find the container with id 5c66adee0f128375997a9257aa201e71f43261e938e365962240d9567fe219d6 Oct 30 06:06:23.876189 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:23.876159 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" event={"ID":"9b95207c-4d87-4e10-809a-7c4d4fbefe6e","Type":"ContainerStarted","Data":"b01035d0dcee862f719ff2b8c0360dd1c8df2a5839b83d05fea8a167ea0246d2"} Oct 30 06:06:23.876676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:23.876205 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" event={"ID":"9b95207c-4d87-4e10-809a-7c4d4fbefe6e","Type":"ContainerStarted","Data":"5c66adee0f128375997a9257aa201e71f43261e938e365962240d9567fe219d6"} Oct 30 06:06:23.876676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:23.876341 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" Oct 30 06:06:23.918014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:23.917955 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" podStartSLOduration=1.9179407450000001 podStartE2EDuration="1.917940745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:23.915898081 +0000 UTC m=+750.424917970" watchObservedRunningTime="2026-10-30 06:06:23.917940745 +0000 UTC m=+750.426960634" Oct 30 06:06:24.785968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:24.785931 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:26.780114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:26.780088 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:26.842605 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:26.842581 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" Oct 30 06:06:26.842804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:26.842653 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" Oct 30 06:06:26.842888 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:26.842815 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" Oct 30 06:06:27.847436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:27.847406 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" Oct 30 06:06:28.665791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:28.665757 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:30.698108 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:30.698077 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:30.837833 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:30.837805 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:30.969654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:30.969584 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp"] Oct 30 06:06:30.975259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:30.975241 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" Oct 30 06:06:30.991904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:30.991874 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:06:30.996538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:30.996516 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:31.003961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:31.003942 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:06:31.006472 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:31.006451 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp"] Oct 30 06:06:31.156390 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:31.156363 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snghm\" (UniqueName: \"kubernetes.io/projected/3404c041-152b-488f-9fcc-89efb534767d-kube-api-access-snghm\") pod \"maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp\" (UID: \"3404c041-152b-488f-9fcc-89efb534767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" Oct 30 06:06:31.156548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:31.156416 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3404c041-152b-488f-9fcc-89efb534767d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp\" (UID: \"3404c041-152b-488f-9fcc-89efb534767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" Oct 30 06:06:31.156616 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:31.156573 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3404c041-152b-488f-9fcc-89efb534767d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp\" (UID: \"3404c041-152b-488f-9fcc-89efb534767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" Oct 30 06:06:31.257822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:31.257752 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3404c041-152b-488f-9fcc-89efb534767d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp\" (UID: \"3404c041-152b-488f-9fcc-89efb534767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" Oct 30 06:06:31.257975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:31.257847 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3404c041-152b-488f-9fcc-89efb534767d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp\" (UID: \"3404c041-152b-488f-9fcc-89efb534767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" Oct 30 06:06:31.257975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:31.257906 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snghm\" (UniqueName: \"kubernetes.io/projected/3404c041-152b-488f-9fcc-89efb534767d-kube-api-access-snghm\") pod \"maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp\" (UID: \"3404c041-152b-488f-9fcc-89efb534767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" Oct 30 06:06:31.258457 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:31.258438 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3404c041-152b-488f-9fcc-89efb534767d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp\" (UID: \"3404c041-152b-488f-9fcc-89efb534767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" Oct 30 06:06:31.260313 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:31.260292 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3404c041-152b-488f-9fcc-89efb534767d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp\" (UID: \"3404c041-152b-488f-9fcc-89efb534767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" Oct 30 06:06:31.302308 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:31.302281 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snghm\" (UniqueName: \"kubernetes.io/projected/3404c041-152b-488f-9fcc-89efb534767d-kube-api-access-snghm\") pod \"maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp\" (UID: \"3404c041-152b-488f-9fcc-89efb534767d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" Oct 30 06:06:31.588755 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:31.588681 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" Oct 30 06:06:31.893111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:31.893080 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp"] Oct 30 06:06:31.897249 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:06:31.897194 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3404c041_152b_488f_9fcc_89efb534767d.slice/crio-95a527139bd5b6c634eed4d1b3b0c97b515716abb555d23ebacdec9dabe5e1ac WatchSource:0}: Error finding container 95a527139bd5b6c634eed4d1b3b0c97b515716abb555d23ebacdec9dabe5e1ac: Status 404 returned error can't find the container with id 95a527139bd5b6c634eed4d1b3b0c97b515716abb555d23ebacdec9dabe5e1ac Oct 30 06:06:31.915734 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:31.915704 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" event={"ID":"3404c041-152b-488f-9fcc-89efb534767d","Type":"ContainerStarted","Data":"95a527139bd5b6c634eed4d1b3b0c97b515716abb555d23ebacdec9dabe5e1ac"} Oct 30 06:06:32.851497 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:32.851463 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:34.140875 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:06:34.140848 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:06:34.768824 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:34.768796 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:34.885860 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:34.885837 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" Oct 30 06:06:34.933487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:34.933456 6999 generic.go:356] "Generic (PLEG): container finished" podID="3404c041-152b-488f-9fcc-89efb534767d" containerID="38fab71062104b8b32b17177e1bf9870e54683eb37a3a171749dba24d48cea3a" exitCode=0 Oct 30 06:06:34.933626 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:34.933524 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" event={"ID":"3404c041-152b-488f-9fcc-89efb534767d","Type":"ContainerDied","Data":"38fab71062104b8b32b17177e1bf9870e54683eb37a3a171749dba24d48cea3a"} Oct 30 06:06:36.075275 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.075254 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" Oct 30 06:06:36.102558 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.102534 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3404c041-152b-488f-9fcc-89efb534767d-kube-api-access-snghm\" (UniqueName: \"kubernetes.io/projected/3404c041-152b-488f-9fcc-89efb534767d-kube-api-access-snghm\") pod \"3404c041-152b-488f-9fcc-89efb534767d\" (UID: \"3404c041-152b-488f-9fcc-89efb534767d\") " Oct 30 06:06:36.102669 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.102576 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3404c041-152b-488f-9fcc-89efb534767d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3404c041-152b-488f-9fcc-89efb534767d-maas-api-openshift-ca\") pod \"3404c041-152b-488f-9fcc-89efb534767d\" (UID: \"3404c041-152b-488f-9fcc-89efb534767d\") " Oct 30 06:06:36.102669 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.102619 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3404c041-152b-488f-9fcc-89efb534767d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3404c041-152b-488f-9fcc-89efb534767d-maas-api-xks-ca\") pod \"3404c041-152b-488f-9fcc-89efb534767d\" (UID: \"3404c041-152b-488f-9fcc-89efb534767d\") " Oct 30 06:06:36.102980 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.102958 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3404c041-152b-488f-9fcc-89efb534767d-maas-api-openshift-ca" pod "3404c041-152b-488f-9fcc-89efb534767d" (UID: "3404c041-152b-488f-9fcc-89efb534767d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:36.104795 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.104771 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3404c041-152b-488f-9fcc-89efb534767d-kube-api-access-snghm" pod "3404c041-152b-488f-9fcc-89efb534767d" (UID: "3404c041-152b-488f-9fcc-89efb534767d"). InnerVolumeSpecName "kube-api-access-snghm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:36.104872 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.104794 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3404c041-152b-488f-9fcc-89efb534767d-maas-api-xks-ca" pod "3404c041-152b-488f-9fcc-89efb534767d" (UID: "3404c041-152b-488f-9fcc-89efb534767d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:36.204224 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.204128 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3404c041-152b-488f-9fcc-89efb534767d-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:06:36.204224 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.204162 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3404c041-152b-488f-9fcc-89efb534767d-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:06:36.204224 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.204172 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snghm\" (UniqueName: \"kubernetes.io/projected/3404c041-152b-488f-9fcc-89efb534767d-kube-api-access-snghm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:06:36.204427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.204346 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx"] Oct 30 06:06:36.414780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.414745 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:36.607696 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.607617 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:36.848722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.848697 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:36.947414 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.947374 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" event={"ID":"3404c041-152b-488f-9fcc-89efb534767d","Type":"ContainerDied","Data":"95a527139bd5b6c634eed4d1b3b0c97b515716abb555d23ebacdec9dabe5e1ac"} Oct 30 06:06:36.947732 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.947695 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95a527139bd5b6c634eed4d1b3b0c97b515716abb555d23ebacdec9dabe5e1ac" Oct 30 06:06:36.947877 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:36.947413 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp" Oct 30 06:06:38.819037 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:38.819007 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:40.858751 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:40.858722 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:41.204999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.204919 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" podUID="0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8" containerName="payload-processing" containerID="cri-o://1f4f199827b94c005e5e802d37edd875e7153cf0f25154d6fde569bd55b72d98" gracePeriod=25 Oct 30 06:06:41.441986 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.441964 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff"] Oct 30 06:06:41.458726 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.458664 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" Oct 30 06:06:41.549593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.549558 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8-kube-api-access-fw5ck\" (UniqueName: \"kubernetes.io/projected/0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8-kube-api-access-fw5ck\") pod \"0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8\" (UID: \"0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8\") " Oct 30 06:06:41.549789 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.549718 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8-plugins-config-volume\") pod \"0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8\" (UID: \"0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8\") " Oct 30 06:06:41.550095 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.550074 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8-plugins-config-volume" pod "0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8" (UID: "0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:41.551865 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.551834 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8-kube-api-access-fw5ck" pod "0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8" (UID: "0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8"). InnerVolumeSpecName "kube-api-access-fw5ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:41.623051 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.623026 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:41.651196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.651168 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw5ck\" (UniqueName: \"kubernetes.io/projected/0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8-kube-api-access-fw5ck\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:06:41.651196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.651200 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:06:41.772015 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.771935 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:41.972132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.972101 6999 generic.go:356] "Generic (PLEG): container finished" podID="0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8" containerID="1f4f199827b94c005e5e802d37edd875e7153cf0f25154d6fde569bd55b72d98" exitCode=0 Oct 30 06:06:41.972557 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.972148 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" Oct 30 06:06:41.972557 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.972151 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" event={"ID":"0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8","Type":"ContainerDied","Data":"1f4f199827b94c005e5e802d37edd875e7153cf0f25154d6fde569bd55b72d98"} Oct 30 06:06:41.972557 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.972276 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx" event={"ID":"0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8","Type":"ContainerDied","Data":"9285a997d64c542c2adf0d90188e0b7e9e6d7ca6d1aed6aa950294a7d54d2f9c"} Oct 30 06:06:41.972557 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.972304 6999 scope.go:122] "RemoveContainer" containerID="1f4f199827b94c005e5e802d37edd875e7153cf0f25154d6fde569bd55b72d98" Oct 30 06:06:41.981142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.981124 6999 scope.go:122] "RemoveContainer" containerID="1f4f199827b94c005e5e802d37edd875e7153cf0f25154d6fde569bd55b72d98" Oct 30 06:06:41.981425 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:06:41.981405 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4f199827b94c005e5e802d37edd875e7153cf0f25154d6fde569bd55b72d98\": container with ID starting with 1f4f199827b94c005e5e802d37edd875e7153cf0f25154d6fde569bd55b72d98 not found: ID does not exist" containerID="1f4f199827b94c005e5e802d37edd875e7153cf0f25154d6fde569bd55b72d98" Oct 30 06:06:41.981491 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:41.981434 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4f199827b94c005e5e802d37edd875e7153cf0f25154d6fde569bd55b72d98"} err="failed to get container status \"1f4f199827b94c005e5e802d37edd875e7153cf0f25154d6fde569bd55b72d98\": rpc error: code = NotFound desc = could not find container \"1f4f199827b94c005e5e802d37edd875e7153cf0f25154d6fde569bd55b72d98\": container with ID starting with 1f4f199827b94c005e5e802d37edd875e7153cf0f25154d6fde569bd55b72d98 not found: ID does not exist" Oct 30 06:06:42.021774 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:42.021754 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx"] Oct 30 06:06:42.033720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:42.033666 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-798193ae-66b87c8994-w7jsx"] Oct 30 06:06:42.221152 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:42.221120 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds"] Oct 30 06:06:42.412994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:42.412962 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:42.552382 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:42.552352 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:42.802853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:42.802776 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:43.607233 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:43.607185 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8" path="/var/lib/kubelet/pods/0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8/volumes" Oct 30 06:06:44.739412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:44.739371 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:45.926024 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:45.925989 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r"] Oct 30 06:06:45.926642 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:45.926621 6999 cpu_manager.go:414] "containerMap: removing container" podUID="3404c041-152b-488f-9fcc-89efb534767d" containerName="revoke-keys" Oct 30 06:06:45.926731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:45.926644 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3404c041-152b-488f-9fcc-89efb534767d" containerName="revoke-keys" Oct 30 06:06:45.926731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:45.926691 6999 cpu_manager.go:414] "containerMap: removing container" podUID="0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8" containerName="payload-processing" Oct 30 06:06:45.926731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:45.926700 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8" containerName="payload-processing" Oct 30 06:06:45.926878 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:45.926799 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d1edbe2-0ec3-4ea0-a85b-e77480bc43d8" containerName="payload-processing" Oct 30 06:06:45.926878 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:45.926818 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="3404c041-152b-488f-9fcc-89efb534767d" containerName="revoke-keys" Oct 30 06:06:45.931342 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:45.931317 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" Oct 30 06:06:45.949563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:45.949540 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r"] Oct 30 06:06:45.959333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:45.959315 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:06:45.959444 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:45.959375 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:06:45.989357 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:45.989335 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktwsv\" (UniqueName: \"kubernetes.io/projected/3247546c-d440-468c-b976-ab1c2c68d90e-kube-api-access-ktwsv\") pod \"maas-api-revoke-keys-e2e-derive-3627d700-qrk4r\" (UID: \"3247546c-d440-468c-b976-ab1c2c68d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" Oct 30 06:06:45.989495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:45.989392 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3247546c-d440-468c-b976-ab1c2c68d90e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3627d700-qrk4r\" (UID: \"3247546c-d440-468c-b976-ab1c2c68d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" Oct 30 06:06:45.989495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:45.989449 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3247546c-d440-468c-b976-ab1c2c68d90e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3627d700-qrk4r\" (UID: \"3247546c-d440-468c-b976-ab1c2c68d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" Oct 30 06:06:46.090035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.090009 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3247546c-d440-468c-b976-ab1c2c68d90e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3627d700-qrk4r\" (UID: \"3247546c-d440-468c-b976-ab1c2c68d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" Oct 30 06:06:46.090177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.090071 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktwsv\" (UniqueName: \"kubernetes.io/projected/3247546c-d440-468c-b976-ab1c2c68d90e-kube-api-access-ktwsv\") pod \"maas-api-revoke-keys-e2e-derive-3627d700-qrk4r\" (UID: \"3247546c-d440-468c-b976-ab1c2c68d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" Oct 30 06:06:46.090177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.090145 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3247546c-d440-468c-b976-ab1c2c68d90e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3627d700-qrk4r\" (UID: \"3247546c-d440-468c-b976-ab1c2c68d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" Oct 30 06:06:46.090764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.090743 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3247546c-d440-468c-b976-ab1c2c68d90e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3627d700-qrk4r\" (UID: \"3247546c-d440-468c-b976-ab1c2c68d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" Oct 30 06:06:46.092636 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.092612 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3247546c-d440-468c-b976-ab1c2c68d90e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3627d700-qrk4r\" (UID: \"3247546c-d440-468c-b976-ab1c2c68d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" Oct 30 06:06:46.113377 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.113356 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktwsv\" (UniqueName: \"kubernetes.io/projected/3247546c-d440-468c-b976-ab1c2c68d90e-kube-api-access-ktwsv\") pod \"maas-api-revoke-keys-e2e-derive-3627d700-qrk4r\" (UID: \"3247546c-d440-468c-b976-ab1c2c68d90e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" Oct 30 06:06:46.244779 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.244713 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" Oct 30 06:06:46.352535 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.352187 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:46.442904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.442847 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" podUID="2511a9a4-7e0d-4006-a550-2a05b060566f" containerName="payload-processing" containerID="cri-o://4ba8d51e6f48bc7276552c91bb4c1fcb1e278ea61328d2a60bf7795e1dbf3570" gracePeriod=25 Oct 30 06:06:46.563552 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.563418 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r"] Oct 30 06:06:46.568456 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:06:46.568424 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3247546c_d440_468c_b976_ab1c2c68d90e.slice/crio-a3bc8d33e3845fddfc4aa698f2d952c1bd65db407afd6f3649a96719cc62b6ad WatchSource:0}: Error finding container a3bc8d33e3845fddfc4aa698f2d952c1bd65db407afd6f3649a96719cc62b6ad: Status 404 returned error can't find the container with id a3bc8d33e3845fddfc4aa698f2d952c1bd65db407afd6f3649a96719cc62b6ad Oct 30 06:06:46.720183 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.720157 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" Oct 30 06:06:46.775881 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.775855 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:46.798052 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.798023 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2511a9a4-7e0d-4006-a550-2a05b060566f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2511a9a4-7e0d-4006-a550-2a05b060566f-plugins-config-volume\") pod \"2511a9a4-7e0d-4006-a550-2a05b060566f\" (UID: \"2511a9a4-7e0d-4006-a550-2a05b060566f\") " Oct 30 06:06:46.798195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.798074 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2511a9a4-7e0d-4006-a550-2a05b060566f-kube-api-access-p448d\" (UniqueName: \"kubernetes.io/projected/2511a9a4-7e0d-4006-a550-2a05b060566f-kube-api-access-p448d\") pod \"2511a9a4-7e0d-4006-a550-2a05b060566f\" (UID: \"2511a9a4-7e0d-4006-a550-2a05b060566f\") " Oct 30 06:06:46.798610 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.798589 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2511a9a4-7e0d-4006-a550-2a05b060566f-plugins-config-volume" pod "2511a9a4-7e0d-4006-a550-2a05b060566f" (UID: "2511a9a4-7e0d-4006-a550-2a05b060566f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:46.800525 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.800502 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2511a9a4-7e0d-4006-a550-2a05b060566f-kube-api-access-p448d" pod "2511a9a4-7e0d-4006-a550-2a05b060566f" (UID: "2511a9a4-7e0d-4006-a550-2a05b060566f"). InnerVolumeSpecName "kube-api-access-p448d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:46.899584 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.899559 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p448d\" (UniqueName: \"kubernetes.io/projected/2511a9a4-7e0d-4006-a550-2a05b060566f-kube-api-access-p448d\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:06:46.899720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.899589 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2511a9a4-7e0d-4006-a550-2a05b060566f-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:06:46.915470 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:46.915445 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:47.000114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.000076 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" event={"ID":"3247546c-d440-468c-b976-ab1c2c68d90e","Type":"ContainerStarted","Data":"6929b8fb11437af442a6de8b9dd68e8df559187288e8f798ba305872ec1c0ed1"} Oct 30 06:06:47.000507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.000145 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" event={"ID":"3247546c-d440-468c-b976-ab1c2c68d90e","Type":"ContainerStarted","Data":"a3bc8d33e3845fddfc4aa698f2d952c1bd65db407afd6f3649a96719cc62b6ad"} Oct 30 06:06:47.001538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.001509 6999 generic.go:356] "Generic (PLEG): container finished" podID="2511a9a4-7e0d-4006-a550-2a05b060566f" containerID="4ba8d51e6f48bc7276552c91bb4c1fcb1e278ea61328d2a60bf7795e1dbf3570" exitCode=0 Oct 30 06:06:47.001659 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.001545 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" Oct 30 06:06:47.001659 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.001560 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" event={"ID":"2511a9a4-7e0d-4006-a550-2a05b060566f","Type":"ContainerDied","Data":"4ba8d51e6f48bc7276552c91bb4c1fcb1e278ea61328d2a60bf7795e1dbf3570"} Oct 30 06:06:47.001659 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.001595 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff" event={"ID":"2511a9a4-7e0d-4006-a550-2a05b060566f","Type":"ContainerDied","Data":"c7fb60b9de6af3f5875093843b544ba6ee74ff62f3596691cefed36448da53f2"} Oct 30 06:06:47.001659 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.001623 6999 scope.go:122] "RemoveContainer" containerID="4ba8d51e6f48bc7276552c91bb4c1fcb1e278ea61328d2a60bf7795e1dbf3570" Oct 30 06:06:47.011254 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.011234 6999 scope.go:122] "RemoveContainer" containerID="4ba8d51e6f48bc7276552c91bb4c1fcb1e278ea61328d2a60bf7795e1dbf3570" Oct 30 06:06:47.011495 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:06:47.011475 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba8d51e6f48bc7276552c91bb4c1fcb1e278ea61328d2a60bf7795e1dbf3570\": container with ID starting with 4ba8d51e6f48bc7276552c91bb4c1fcb1e278ea61328d2a60bf7795e1dbf3570 not found: ID does not exist" containerID="4ba8d51e6f48bc7276552c91bb4c1fcb1e278ea61328d2a60bf7795e1dbf3570" Oct 30 06:06:47.011579 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.011500 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba8d51e6f48bc7276552c91bb4c1fcb1e278ea61328d2a60bf7795e1dbf3570"} err="failed to get container status \"4ba8d51e6f48bc7276552c91bb4c1fcb1e278ea61328d2a60bf7795e1dbf3570\": rpc error: code = NotFound desc = could not find container \"4ba8d51e6f48bc7276552c91bb4c1fcb1e278ea61328d2a60bf7795e1dbf3570\": container with ID starting with 4ba8d51e6f48bc7276552c91bb4c1fcb1e278ea61328d2a60bf7795e1dbf3570 not found: ID does not exist" Oct 30 06:06:47.050245 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.050181 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" podStartSLOduration=2.050170395 podStartE2EDuration="2.050170395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:47.049977944 +0000 UTC m=+773.558997831" watchObservedRunningTime="2026-10-30 06:06:47.050170395 +0000 UTC m=+773.559190282" Oct 30 06:06:47.071479 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.071459 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:47.097864 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.097844 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff"] Oct 30 06:06:47.100072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.100021 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-c45a2ff8-875d9b65b-k9mff"] Oct 30 06:06:47.221897 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.221855 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" podUID="f105f460-ecf2-4ebd-88b7-e19e979c5cfb" containerName="payload-processing" containerID="cri-o://b87d5f1e739dcce537832028ffa81fa24fc2e284df8760bc5abe6bd0e69f3f84" gracePeriod=25 Oct 30 06:06:47.271895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.271869 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:47.429422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.429390 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:47.461152 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.461122 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp"] Oct 30 06:06:47.464410 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.464334 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c45a2ff8-fttpp"] Oct 30 06:06:47.481255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.481229 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" Oct 30 06:06:47.506048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.506024 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f105f460-ecf2-4ebd-88b7-e19e979c5cfb-kube-api-access-nxmjw\" (UniqueName: \"kubernetes.io/projected/f105f460-ecf2-4ebd-88b7-e19e979c5cfb-kube-api-access-nxmjw\") pod \"f105f460-ecf2-4ebd-88b7-e19e979c5cfb\" (UID: \"f105f460-ecf2-4ebd-88b7-e19e979c5cfb\") " Oct 30 06:06:47.506165 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.506103 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f105f460-ecf2-4ebd-88b7-e19e979c5cfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f105f460-ecf2-4ebd-88b7-e19e979c5cfb-plugins-config-volume\") pod \"f105f460-ecf2-4ebd-88b7-e19e979c5cfb\" (UID: \"f105f460-ecf2-4ebd-88b7-e19e979c5cfb\") " Oct 30 06:06:47.506522 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.506497 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f105f460-ecf2-4ebd-88b7-e19e979c5cfb-plugins-config-volume" pod "f105f460-ecf2-4ebd-88b7-e19e979c5cfb" (UID: "f105f460-ecf2-4ebd-88b7-e19e979c5cfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:47.508365 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.508345 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f105f460-ecf2-4ebd-88b7-e19e979c5cfb-kube-api-access-nxmjw" pod "f105f460-ecf2-4ebd-88b7-e19e979c5cfb" (UID: "f105f460-ecf2-4ebd-88b7-e19e979c5cfb"). InnerVolumeSpecName "kube-api-access-nxmjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:47.576769 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.576738 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:47.607058 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.607021 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f105f460-ecf2-4ebd-88b7-e19e979c5cfb-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:06:47.607058 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.607058 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxmjw\" (UniqueName: \"kubernetes.io/projected/f105f460-ecf2-4ebd-88b7-e19e979c5cfb-kube-api-access-nxmjw\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:06:47.634472 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.632262 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2511a9a4-7e0d-4006-a550-2a05b060566f" path="/var/lib/kubelet/pods/2511a9a4-7e0d-4006-a550-2a05b060566f/volumes" Oct 30 06:06:47.634472 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.632732 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3404c041-152b-488f-9fcc-89efb534767d" path="/var/lib/kubelet/pods/3404c041-152b-488f-9fcc-89efb534767d/volumes" Oct 30 06:06:47.740887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.740847 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:47.991116 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:47.991028 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:48.007657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.007631 6999 generic.go:356] "Generic (PLEG): container finished" podID="f105f460-ecf2-4ebd-88b7-e19e979c5cfb" containerID="b87d5f1e739dcce537832028ffa81fa24fc2e284df8760bc5abe6bd0e69f3f84" exitCode=0 Oct 30 06:06:48.008101 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.007676 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" Oct 30 06:06:48.008101 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.007711 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" event={"ID":"f105f460-ecf2-4ebd-88b7-e19e979c5cfb","Type":"ContainerDied","Data":"b87d5f1e739dcce537832028ffa81fa24fc2e284df8760bc5abe6bd0e69f3f84"} Oct 30 06:06:48.008101 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.007764 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds" event={"ID":"f105f460-ecf2-4ebd-88b7-e19e979c5cfb","Type":"ContainerDied","Data":"6a3ad3324badb307a3c4bde5b0bd1674100e0c58db9bf7d9e0ade818076ea57e"} Oct 30 06:06:48.008101 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.007796 6999 scope.go:122] "RemoveContainer" containerID="b87d5f1e739dcce537832028ffa81fa24fc2e284df8760bc5abe6bd0e69f3f84" Oct 30 06:06:48.009976 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.009948 6999 generic.go:356] "Generic (PLEG): container finished" podID="3247546c-d440-468c-b976-ab1c2c68d90e" containerID="6929b8fb11437af442a6de8b9dd68e8df559187288e8f798ba305872ec1c0ed1" exitCode=0 Oct 30 06:06:48.010134 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.010112 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" event={"ID":"3247546c-d440-468c-b976-ab1c2c68d90e","Type":"ContainerDied","Data":"6929b8fb11437af442a6de8b9dd68e8df559187288e8f798ba305872ec1c0ed1"} Oct 30 06:06:48.017368 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.017349 6999 scope.go:122] "RemoveContainer" containerID="b87d5f1e739dcce537832028ffa81fa24fc2e284df8760bc5abe6bd0e69f3f84" Oct 30 06:06:48.017663 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:06:48.017645 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87d5f1e739dcce537832028ffa81fa24fc2e284df8760bc5abe6bd0e69f3f84\": container with ID starting with b87d5f1e739dcce537832028ffa81fa24fc2e284df8760bc5abe6bd0e69f3f84 not found: ID does not exist" containerID="b87d5f1e739dcce537832028ffa81fa24fc2e284df8760bc5abe6bd0e69f3f84" Oct 30 06:06:48.017734 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.017674 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87d5f1e739dcce537832028ffa81fa24fc2e284df8760bc5abe6bd0e69f3f84"} err="failed to get container status \"b87d5f1e739dcce537832028ffa81fa24fc2e284df8760bc5abe6bd0e69f3f84\": rpc error: code = NotFound desc = could not find container \"b87d5f1e739dcce537832028ffa81fa24fc2e284df8760bc5abe6bd0e69f3f84\": container with ID starting with b87d5f1e739dcce537832028ffa81fa24fc2e284df8760bc5abe6bd0e69f3f84 not found: ID does not exist" Oct 30 06:06:48.068860 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.068837 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds"] Oct 30 06:06:48.074436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.074415 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-9c94fbc2-9c7c645f4-dnpds"] Oct 30 06:06:48.164010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.163973 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:48.325564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.325476 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:48.573533 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.573500 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89"] Oct 30 06:06:48.574207 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.574182 6999 cpu_manager.go:414] "containerMap: removing container" podUID="2511a9a4-7e0d-4006-a550-2a05b060566f" containerName="payload-processing" Oct 30 06:06:48.574207 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.574206 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2511a9a4-7e0d-4006-a550-2a05b060566f" containerName="payload-processing" Oct 30 06:06:48.574407 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.574258 6999 cpu_manager.go:414] "containerMap: removing container" podUID="f105f460-ecf2-4ebd-88b7-e19e979c5cfb" containerName="payload-processing" Oct 30 06:06:48.574407 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.574264 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f105f460-ecf2-4ebd-88b7-e19e979c5cfb" containerName="payload-processing" Oct 30 06:06:48.574407 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.574355 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="f105f460-ecf2-4ebd-88b7-e19e979c5cfb" containerName="payload-processing" Oct 30 06:06:48.574407 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.574373 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="2511a9a4-7e0d-4006-a550-2a05b060566f" containerName="payload-processing" Oct 30 06:06:48.577609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.577563 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.579687 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.579666 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:48.602106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.602041 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89"] Oct 30 06:06:48.611910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.611890 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8d6349-openshift-default-dockercfg-xs2qm\"" Oct 30 06:06:48.616679 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.616656 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.616802 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.616699 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.616802 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.616734 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.616888 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.616834 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.616888 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.616855 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.616991 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.616891 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvtr8\" (UniqueName: \"kubernetes.io/projected/bbd16456-e1e8-4295-8d4a-3045f9e20db1-kube-api-access-cvtr8\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.617045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.617015 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.617106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.617052 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.617106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.617089 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.718387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.718347 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.718387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.718384 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.718633 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.718555 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtr8\" (UniqueName: \"kubernetes.io/projected/bbd16456-e1e8-4295-8d4a-3045f9e20db1-kube-api-access-cvtr8\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.718633 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.718624 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.718749 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.718650 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.718749 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.718719 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.718851 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.718774 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.718909 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.718852 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.719228 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.719155 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.719228 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.719164 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.719596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.719565 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.719733 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.719640 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.720049 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.720026 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.720049 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.720044 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.723132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.723108 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.723346 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.723325 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.738016 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.737992 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:48.748672 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.748649 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.749226 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.749191 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtr8\" (UniqueName: \"kubernetes.io/projected/bbd16456-e1e8-4295-8d4a-3045f9e20db1-kube-api-access-cvtr8\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:48.792078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.792050 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g"] Oct 30 06:06:48.891805 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.891774 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:48.899737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:48.899710 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:49.057683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.057651 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:49.169330 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.169306 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" Oct 30 06:06:49.225010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.224980 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3247546c-d440-468c-b976-ab1c2c68d90e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3247546c-d440-468c-b976-ab1c2c68d90e-maas-api-openshift-ca\") pod \"3247546c-d440-468c-b976-ab1c2c68d90e\" (UID: \"3247546c-d440-468c-b976-ab1c2c68d90e\") " Oct 30 06:06:49.225189 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.225113 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3247546c-d440-468c-b976-ab1c2c68d90e-kube-api-access-ktwsv\" (UniqueName: \"kubernetes.io/projected/3247546c-d440-468c-b976-ab1c2c68d90e-kube-api-access-ktwsv\") pod \"3247546c-d440-468c-b976-ab1c2c68d90e\" (UID: \"3247546c-d440-468c-b976-ab1c2c68d90e\") " Oct 30 06:06:49.225189 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.225149 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3247546c-d440-468c-b976-ab1c2c68d90e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3247546c-d440-468c-b976-ab1c2c68d90e-maas-api-xks-ca\") pod \"3247546c-d440-468c-b976-ab1c2c68d90e\" (UID: \"3247546c-d440-468c-b976-ab1c2c68d90e\") " Oct 30 06:06:49.225409 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.225388 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3247546c-d440-468c-b976-ab1c2c68d90e-maas-api-openshift-ca" pod "3247546c-d440-468c-b976-ab1c2c68d90e" (UID: "3247546c-d440-468c-b976-ab1c2c68d90e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:49.225507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.225493 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3247546c-d440-468c-b976-ab1c2c68d90e-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:06:49.227453 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.227423 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3247546c-d440-468c-b976-ab1c2c68d90e-maas-api-xks-ca" pod "3247546c-d440-468c-b976-ab1c2c68d90e" (UID: "3247546c-d440-468c-b976-ab1c2c68d90e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:49.227549 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.227453 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3247546c-d440-468c-b976-ab1c2c68d90e-kube-api-access-ktwsv" pod "3247546c-d440-468c-b976-ab1c2c68d90e" (UID: "3247546c-d440-468c-b976-ab1c2c68d90e"). InnerVolumeSpecName "kube-api-access-ktwsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:49.233126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.233105 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89"] Oct 30 06:06:49.236438 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:06:49.236407 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd16456_e1e8_4295_8d4a_3045f9e20db1.slice/crio-8a1e66d7e0628ac950d0f0ba2885cb6886a354bd9e2e585bfbc5cfd2a29c338b WatchSource:0}: Error finding container 8a1e66d7e0628ac950d0f0ba2885cb6886a354bd9e2e585bfbc5cfd2a29c338b: Status 404 returned error can't find the container with id 8a1e66d7e0628ac950d0f0ba2885cb6886a354bd9e2e585bfbc5cfd2a29c338b Oct 30 06:06:49.238899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.238868 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:49.239001 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.238933 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:49.239001 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.238961 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:49.240901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.240877 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:49.326814 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.326790 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktwsv\" (UniqueName: \"kubernetes.io/projected/3247546c-d440-468c-b976-ab1c2c68d90e-kube-api-access-ktwsv\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:06:49.326814 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.326815 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3247546c-d440-468c-b976-ab1c2c68d90e-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:06:49.397586 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.397519 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:49.608046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:49.608019 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f105f460-ecf2-4ebd-88b7-e19e979c5cfb" path="/var/lib/kubelet/pods/f105f460-ecf2-4ebd-88b7-e19e979c5cfb/volumes" Oct 30 06:06:50.024829 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:50.024800 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" Oct 30 06:06:50.025011 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:50.024803 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r" event={"ID":"3247546c-d440-468c-b976-ab1c2c68d90e","Type":"ContainerDied","Data":"a3bc8d33e3845fddfc4aa698f2d952c1bd65db407afd6f3649a96719cc62b6ad"} Oct 30 06:06:50.025011 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:50.024938 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3bc8d33e3845fddfc4aa698f2d952c1bd65db407afd6f3649a96719cc62b6ad" Oct 30 06:06:50.026610 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:50.026582 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" event={"ID":"bbd16456-e1e8-4295-8d4a-3045f9e20db1","Type":"ContainerStarted","Data":"56a7548fb3a375198e80cf48bebce5cecb927a3f536d32933e87ef09775b8f06"} Oct 30 06:06:50.026761 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:50.026627 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" event={"ID":"bbd16456-e1e8-4295-8d4a-3045f9e20db1","Type":"ContainerStarted","Data":"8a1e66d7e0628ac950d0f0ba2885cb6886a354bd9e2e585bfbc5cfd2a29c338b"} Oct 30 06:06:50.095590 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:50.095533 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" podStartSLOduration=2.095512005 podStartE2EDuration="2.095512005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:50.082203426 +0000 UTC m=+776.591223335" watchObservedRunningTime="2026-10-30 06:06:50.095512005 +0000 UTC m=+776.604531893" Oct 30 06:06:50.098692 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:50.098662 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:50.259224 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:50.259171 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:50.774999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:50.774969 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:50.900051 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:50.900028 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:50.904780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:50.904760 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:51.032473 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.032390 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:51.033282 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.033262 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:06:51.225732 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.225698 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26"] Oct 30 06:06:51.226482 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.226461 6999 cpu_manager.go:414] "containerMap: removing container" podUID="3247546c-d440-468c-b976-ab1c2c68d90e" containerName="revoke-keys" Oct 30 06:06:51.226544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.226488 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3247546c-d440-468c-b976-ab1c2c68d90e" containerName="revoke-keys" Oct 30 06:06:51.226649 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.226636 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="3247546c-d440-468c-b976-ab1c2c68d90e" containerName="revoke-keys" Oct 30 06:06:51.230845 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.230821 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.255876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.255851 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-eaee88-openshift-default-dockercfg-fmzvg\"" Oct 30 06:06:51.256066 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.256013 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26"] Oct 30 06:06:51.340785 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.340711 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:51.346885 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.346857 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39b2299-705b-4969-8669-1d796017249d-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.347021 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.346920 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.347070 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.347034 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.347114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.347073 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.347114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.347105 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39b2299-705b-4969-8669-1d796017249d-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.347199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.347176 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39b2299-705b-4969-8669-1d796017249d-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.347273 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.347208 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52qtv\" (UniqueName: \"kubernetes.io/projected/d39b2299-705b-4969-8669-1d796017249d-kube-api-access-52qtv\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.347273 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.347257 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.347351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.347291 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.448597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.448557 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39b2299-705b-4969-8669-1d796017249d-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.448798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.448644 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.448798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.448682 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.448798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.448725 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.448798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.448757 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39b2299-705b-4969-8669-1d796017249d-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.449036 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.448815 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39b2299-705b-4969-8669-1d796017249d-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.449036 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.448840 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52qtv\" (UniqueName: \"kubernetes.io/projected/d39b2299-705b-4969-8669-1d796017249d-kube-api-access-52qtv\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.449036 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.448868 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.449036 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.448893 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.449430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.449400 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.449516 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.449444 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.449516 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.449466 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39b2299-705b-4969-8669-1d796017249d-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.449718 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.449681 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.449913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.449895 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.452409 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.452385 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39b2299-705b-4969-8669-1d796017249d-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.452523 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.452424 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.472581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.472551 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39b2299-705b-4969-8669-1d796017249d-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.473017 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.472994 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52qtv\" (UniqueName: \"kubernetes.io/projected/d39b2299-705b-4969-8669-1d796017249d-kube-api-access-52qtv\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.503111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.503078 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:51.547335 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.547308 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:51.872403 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.872372 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26"] Oct 30 06:06:51.880406 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:06:51.880377 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39b2299_705b_4969_8669_1d796017249d.slice/crio-64eadc650cbfd1ddb27d868b631017f8ca0a0f9f7b32256b5966bb9692ad64bf WatchSource:0}: Error finding container 64eadc650cbfd1ddb27d868b631017f8ca0a0f9f7b32256b5966bb9692ad64bf: Status 404 returned error can't find the container with id 64eadc650cbfd1ddb27d868b631017f8ca0a0f9f7b32256b5966bb9692ad64bf Oct 30 06:06:51.882668 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.882636 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:51.882750 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.882706 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:51.882750 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.882734 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:51.923676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:51.923649 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:52.039901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:52.039856 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" event={"ID":"d39b2299-705b-4969-8669-1d796017249d","Type":"ContainerStarted","Data":"449e1fb9e7f6c365c15b98c5dc32a38ff728f3fe7a589caed7506294f69db4a7"} Oct 30 06:06:52.040061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:52.039925 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" event={"ID":"d39b2299-705b-4969-8669-1d796017249d","Type":"ContainerStarted","Data":"64eadc650cbfd1ddb27d868b631017f8ca0a0f9f7b32256b5966bb9692ad64bf"} Oct 30 06:06:52.090410 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:52.089405 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" podStartSLOduration=1.08938649 podStartE2EDuration="1.08938649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:52.086879549 +0000 UTC m=+778.595899438" watchObservedRunningTime="2026-10-30 06:06:52.08938649 +0000 UTC m=+778.598406378" Oct 30 06:06:52.502953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:52.502925 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:52.547637 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:52.547614 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:52.552475 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:52.552456 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:52.649712 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:52.649681 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:52.790618 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:52.790528 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:52.949597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:52.949568 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:53.045577 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:53.045493 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:53.046236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:53.046203 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:06:53.130775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:53.130745 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:53.294796 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:53.294761 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:53.438747 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:53.438720 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:53.793040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:53.793005 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" podUID="86283741-0147-4374-b410-f4dadeefc42e" containerName="payload-pre-processing" containerID="cri-o://3e7abad5261db035f23949ede9446f99c4a071fedb05487e7f06b764bfc79ce9" gracePeriod=25 Oct 30 06:06:54.051842 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:54.051816 6999 generic.go:356] "Generic (PLEG): container finished" podID="86283741-0147-4374-b410-f4dadeefc42e" containerID="3e7abad5261db035f23949ede9446f99c4a071fedb05487e7f06b764bfc79ce9" exitCode=0 Oct 30 06:06:54.051962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:54.051904 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" event={"ID":"86283741-0147-4374-b410-f4dadeefc42e","Type":"ContainerDied","Data":"3e7abad5261db035f23949ede9446f99c4a071fedb05487e7f06b764bfc79ce9"} Oct 30 06:06:54.051962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:54.051954 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" event={"ID":"86283741-0147-4374-b410-f4dadeefc42e","Type":"ContainerDied","Data":"fe5c41fa63151fc207ee6b71a3026932a68cff894612e98a970eabe5131f6f6f"} Oct 30 06:06:54.052046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:54.051974 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe5c41fa63151fc207ee6b71a3026932a68cff894612e98a970eabe5131f6f6f" Oct 30 06:06:54.057830 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:54.057810 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" Oct 30 06:06:54.179110 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:54.179079 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86283741-0147-4374-b410-f4dadeefc42e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86283741-0147-4374-b410-f4dadeefc42e-plugins-config-volume\") pod \"86283741-0147-4374-b410-f4dadeefc42e\" (UID: \"86283741-0147-4374-b410-f4dadeefc42e\") " Oct 30 06:06:54.179300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:54.179246 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86283741-0147-4374-b410-f4dadeefc42e-kube-api-access-dqcl5\" (UniqueName: \"kubernetes.io/projected/86283741-0147-4374-b410-f4dadeefc42e-kube-api-access-dqcl5\") pod \"86283741-0147-4374-b410-f4dadeefc42e\" (UID: \"86283741-0147-4374-b410-f4dadeefc42e\") " Oct 30 06:06:54.179463 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:54.179442 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86283741-0147-4374-b410-f4dadeefc42e-plugins-config-volume" pod "86283741-0147-4374-b410-f4dadeefc42e" (UID: "86283741-0147-4374-b410-f4dadeefc42e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:54.179633 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:54.179614 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86283741-0147-4374-b410-f4dadeefc42e-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:06:54.181410 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:54.181391 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86283741-0147-4374-b410-f4dadeefc42e-kube-api-access-dqcl5" pod "86283741-0147-4374-b410-f4dadeefc42e" (UID: "86283741-0147-4374-b410-f4dadeefc42e"). InnerVolumeSpecName "kube-api-access-dqcl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:54.209999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:54.209974 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:54.281048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:54.280964 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqcl5\" (UniqueName: \"kubernetes.io/projected/86283741-0147-4374-b410-f4dadeefc42e-kube-api-access-dqcl5\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:06:54.571073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:54.570995 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:54.730130 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:54.730096 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:54.877231 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:54.877187 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:55.013876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:55.013841 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:55.056309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:55.056274 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g" Oct 30 06:06:55.106117 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:55.106088 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g"] Oct 30 06:06:55.110229 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:55.110194 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-55140dc4-7cc7c6cbcf-jq79g"] Oct 30 06:06:55.306990 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:55.306922 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:55.467548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:55.467506 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:55.607997 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:55.607926 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86283741-0147-4374-b410-f4dadeefc42e" path="/var/lib/kubelet/pods/86283741-0147-4374-b410-f4dadeefc42e/volumes" Oct 30 06:06:56.267029 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:56.266992 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:56.420939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:56.420910 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:56.915141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:56.915112 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:57.067698 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:57.067671 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:57.202775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:57.202698 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:57.346431 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:57.346302 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:57.705924 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:57.705885 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:58.589071 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:58.589035 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:58.783635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:58.783598 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:59.052584 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:59.052552 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:59.135441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:59.135407 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r"] Oct 30 06:06:59.227555 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:59.227519 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:59.381799 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:59.381774 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:06:59.764160 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:06:59.764083 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:01.060647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:01.060618 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:01.914047 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:01.914017 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:02.103908 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:02.103861 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:03.196996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:03.196964 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:03.331844 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:03.331812 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:04.147354 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:04.147303 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" podUID="9b95207c-4d87-4e10-809a-7c4d4fbefe6e" containerName="payload-processing" containerID="cri-o://b01035d0dcee862f719ff2b8c0360dd1c8df2a5839b83d05fea8a167ea0246d2" gracePeriod=25 Oct 30 06:07:04.164808 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:07:04.164778 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:07:04.459471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:04.459442 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" Oct 30 06:07:04.604911 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:04.604879 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b95207c-4d87-4e10-809a-7c4d4fbefe6e-kube-api-access-qh7qh\" (UniqueName: \"kubernetes.io/projected/9b95207c-4d87-4e10-809a-7c4d4fbefe6e-kube-api-access-qh7qh\") pod \"9b95207c-4d87-4e10-809a-7c4d4fbefe6e\" (UID: \"9b95207c-4d87-4e10-809a-7c4d4fbefe6e\") " Oct 30 06:07:04.605068 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:04.604937 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b95207c-4d87-4e10-809a-7c4d4fbefe6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b95207c-4d87-4e10-809a-7c4d4fbefe6e-plugins-config-volume\") pod \"9b95207c-4d87-4e10-809a-7c4d4fbefe6e\" (UID: \"9b95207c-4d87-4e10-809a-7c4d4fbefe6e\") " Oct 30 06:07:04.605388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:04.605361 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b95207c-4d87-4e10-809a-7c4d4fbefe6e-plugins-config-volume" pod "9b95207c-4d87-4e10-809a-7c4d4fbefe6e" (UID: "9b95207c-4d87-4e10-809a-7c4d4fbefe6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:04.607002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:04.606985 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b95207c-4d87-4e10-809a-7c4d4fbefe6e-kube-api-access-qh7qh" pod "9b95207c-4d87-4e10-809a-7c4d4fbefe6e" (UID: "9b95207c-4d87-4e10-809a-7c4d4fbefe6e"). InnerVolumeSpecName "kube-api-access-qh7qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:04.706171 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:04.706086 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh7qh\" (UniqueName: \"kubernetes.io/projected/9b95207c-4d87-4e10-809a-7c4d4fbefe6e-kube-api-access-qh7qh\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:07:04.706171 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:04.706113 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b95207c-4d87-4e10-809a-7c4d4fbefe6e-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:07:04.858283 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:04.858086 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:05.110857 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.110824 6999 generic.go:356] "Generic (PLEG): container finished" podID="9b95207c-4d87-4e10-809a-7c4d4fbefe6e" containerID="b01035d0dcee862f719ff2b8c0360dd1c8df2a5839b83d05fea8a167ea0246d2" exitCode=0 Oct 30 06:07:05.110999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.110872 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" Oct 30 06:07:05.110999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.110904 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" event={"ID":"9b95207c-4d87-4e10-809a-7c4d4fbefe6e","Type":"ContainerDied","Data":"b01035d0dcee862f719ff2b8c0360dd1c8df2a5839b83d05fea8a167ea0246d2"} Oct 30 06:07:05.110999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.110963 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r" event={"ID":"9b95207c-4d87-4e10-809a-7c4d4fbefe6e","Type":"ContainerDied","Data":"5c66adee0f128375997a9257aa201e71f43261e938e365962240d9567fe219d6"} Oct 30 06:07:05.110999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.110985 6999 scope.go:122] "RemoveContainer" containerID="b01035d0dcee862f719ff2b8c0360dd1c8df2a5839b83d05fea8a167ea0246d2" Oct 30 06:07:05.119842 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.119826 6999 scope.go:122] "RemoveContainer" containerID="b01035d0dcee862f719ff2b8c0360dd1c8df2a5839b83d05fea8a167ea0246d2" Oct 30 06:07:05.120106 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:07:05.120089 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b01035d0dcee862f719ff2b8c0360dd1c8df2a5839b83d05fea8a167ea0246d2\": container with ID starting with b01035d0dcee862f719ff2b8c0360dd1c8df2a5839b83d05fea8a167ea0246d2 not found: ID does not exist" containerID="b01035d0dcee862f719ff2b8c0360dd1c8df2a5839b83d05fea8a167ea0246d2" Oct 30 06:07:05.120153 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.120114 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b01035d0dcee862f719ff2b8c0360dd1c8df2a5839b83d05fea8a167ea0246d2"} err="failed to get container status \"b01035d0dcee862f719ff2b8c0360dd1c8df2a5839b83d05fea8a167ea0246d2\": rpc error: code = NotFound desc = could not find container \"b01035d0dcee862f719ff2b8c0360dd1c8df2a5839b83d05fea8a167ea0246d2\": container with ID starting with b01035d0dcee862f719ff2b8c0360dd1c8df2a5839b83d05fea8a167ea0246d2 not found: ID does not exist" Oct 30 06:07:05.172501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.172473 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r"] Oct 30 06:07:05.185051 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.185022 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-3627d700-796cffd94-gbs8r"] Oct 30 06:07:05.427425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.427344 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:05.555255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.552881 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r"] Oct 30 06:07:05.562298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.562277 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3627d700-qrk4r"] Oct 30 06:07:05.581849 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.581818 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:05.607385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.607356 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3247546c-d440-468c-b976-ab1c2c68d90e" path="/var/lib/kubelet/pods/3247546c-d440-468c-b976-ab1c2c68d90e/volumes" Oct 30 06:07:05.607796 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.607783 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b95207c-4d87-4e10-809a-7c4d4fbefe6e" path="/var/lib/kubelet/pods/9b95207c-4d87-4e10-809a-7c4d4fbefe6e/volumes" Oct 30 06:07:05.854820 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:05.854714 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:06.007263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:06.007196 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:06.652241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:06.650683 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:07.066662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.066393 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74"] Oct 30 06:07:07.068136 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.066963 6999 cpu_manager.go:414] "containerMap: removing container" podUID="86283741-0147-4374-b410-f4dadeefc42e" containerName="payload-pre-processing" Oct 30 06:07:07.068136 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.066986 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86283741-0147-4374-b410-f4dadeefc42e" containerName="payload-pre-processing" Oct 30 06:07:07.068136 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.067020 6999 cpu_manager.go:414] "containerMap: removing container" podUID="9b95207c-4d87-4e10-809a-7c4d4fbefe6e" containerName="payload-processing" Oct 30 06:07:07.068136 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.067027 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b95207c-4d87-4e10-809a-7c4d4fbefe6e" containerName="payload-processing" Oct 30 06:07:07.068136 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.067143 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b95207c-4d87-4e10-809a-7c4d4fbefe6e" containerName="payload-processing" Oct 30 06:07:07.068136 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.067163 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="86283741-0147-4374-b410-f4dadeefc42e" containerName="payload-pre-processing" Oct 30 06:07:07.071557 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.071528 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.099260 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.096427 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8d44dd-openshift-default-dockercfg-nzqwp\"" Oct 30 06:07:07.107617 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.106441 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74"] Oct 30 06:07:07.128853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.128828 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.129023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.128877 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.129023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.128956 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.129023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.128983 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.129194 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.129041 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.129194 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.129075 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.129194 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.129144 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.129194 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.129181 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7htm\" (UniqueName: \"kubernetes.io/projected/0ae13585-2214-4d4c-87ec-0e7ee0b57213-kube-api-access-l7htm\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.129408 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.129233 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.162236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.161685 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:07.230709 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.230671 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.230709 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.230707 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.230946 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.230844 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.230946 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.230901 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7htm\" (UniqueName: \"kubernetes.io/projected/0ae13585-2214-4d4c-87ec-0e7ee0b57213-kube-api-access-l7htm\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.230946 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.230940 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.231103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.231013 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.231103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.231051 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.231103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.231088 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.231328 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.231113 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.231473 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.231450 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.231852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.231567 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.231852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.231700 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.231852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.231791 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.232062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.231939 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.233840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.233815 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.234030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.234011 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.270266 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.260085 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.271930 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.271884 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7htm\" (UniqueName: \"kubernetes.io/projected/0ae13585-2214-4d4c-87ec-0e7ee0b57213-kube-api-access-l7htm\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.357115 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.356795 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:07.387427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.387399 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:07.765043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.765003 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74"] Oct 30 06:07:07.789444 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.789402 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:07.789654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.789630 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:07.789822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.789802 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:07.866578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:07.866546 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:08.032521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:08.032450 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:08.129222 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:08.129175 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" event={"ID":"0ae13585-2214-4d4c-87ec-0e7ee0b57213","Type":"ContainerStarted","Data":"0d7af5f97642789db4d0ead75e21e35538038446d20d52c0654a80161e93cbae"} Oct 30 06:07:08.129374 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:08.129245 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" event={"ID":"0ae13585-2214-4d4c-87ec-0e7ee0b57213","Type":"ContainerStarted","Data":"650117f6fd693ec7fe1fe7d8fc347bda42c7a8f0452709fd12c911364881a67d"} Oct 30 06:07:08.184842 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:08.184788 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" podStartSLOduration=1.184773481 podStartE2EDuration="1.184773481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:08.182864518 +0000 UTC m=+794.691884403" watchObservedRunningTime="2026-10-30 06:07:08.184773481 +0000 UTC m=+794.693793366" Oct 30 06:07:08.202148 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:08.202126 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:08.388133 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:08.388098 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:08.393073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:08.393049 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:09.140246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:09.135440 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:09.155254 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:09.153501 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:10.476807 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:10.476764 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:10.801166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:10.801041 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:10.962242 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:10.954742 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:12.502100 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:12.502064 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:12.662664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:12.662636 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:12.804540 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:12.804447 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:13.181623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:13.181590 6999 generic.go:356] "Generic (PLEG): container finished" podID="bbd16456-e1e8-4295-8d4a-3045f9e20db1" containerID="56a7548fb3a375198e80cf48bebce5cecb927a3f536d32933e87ef09775b8f06" exitCode=0 Oct 30 06:07:13.181802 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:13.181637 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" event={"ID":"bbd16456-e1e8-4295-8d4a-3045f9e20db1","Type":"ContainerDied","Data":"56a7548fb3a375198e80cf48bebce5cecb927a3f536d32933e87ef09775b8f06"} Oct 30 06:07:13.182015 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:13.182002 6999 scope.go:122] "RemoveContainer" containerID="56a7548fb3a375198e80cf48bebce5cecb927a3f536d32933e87ef09775b8f06" Oct 30 06:07:13.183195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:13.183157 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:13.183278 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:13.183263 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:13.183319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:13.183302 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:13.372822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:13.372790 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:13.523965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:13.523873 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:13.680027 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:13.679988 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:14.192683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:14.192655 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" event={"ID":"bbd16456-e1e8-4295-8d4a-3045f9e20db1","Type":"ContainerStarted","Data":"57cbd05bb0588fb3cfe0b0575a8adbc6e9e549533e0a7f0f8956d76e67553dc7"} Oct 30 06:07:14.847278 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:14.846921 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:14.900593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:14.900563 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:07:14.900805 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:14.900781 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 06:07:14.900874 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:14.900853 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" podUID="bbd16456-e1e8-4295-8d4a-3045f9e20db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 06:07:15.132520 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:15.132492 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:15.900082 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:15.900052 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 06:07:15.900513 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:15.900121 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" podUID="bbd16456-e1e8-4295-8d4a-3045f9e20db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 06:07:16.787901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:16.787870 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:16.900994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:16.900958 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 06:07:16.901409 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:16.901023 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" podUID="bbd16456-e1e8-4295-8d4a-3045f9e20db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 06:07:17.900283 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:17.900252 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 06:07:17.900468 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:17.900301 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" podUID="bbd16456-e1e8-4295-8d4a-3045f9e20db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 06:07:18.177094 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:18.177068 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:18.224279 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:18.224246 6999 generic.go:356] "Generic (PLEG): container finished" podID="d39b2299-705b-4969-8669-1d796017249d" containerID="449e1fb9e7f6c365c15b98c5dc32a38ff728f3fe7a589caed7506294f69db4a7" exitCode=0 Oct 30 06:07:18.224419 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:18.224312 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" event={"ID":"d39b2299-705b-4969-8669-1d796017249d","Type":"ContainerDied","Data":"449e1fb9e7f6c365c15b98c5dc32a38ff728f3fe7a589caed7506294f69db4a7"} Oct 30 06:07:18.224686 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:18.224674 6999 scope.go:122] "RemoveContainer" containerID="449e1fb9e7f6c365c15b98c5dc32a38ff728f3fe7a589caed7506294f69db4a7" Oct 30 06:07:18.225868 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:18.225841 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:18.225948 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:18.225920 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:18.225985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:18.225950 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:18.833581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:18.833550 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:18.901924 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:18.901894 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Oct 30 06:07:18.902252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:18.902193 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" podUID="bbd16456-e1e8-4295-8d4a-3045f9e20db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Oct 30 06:07:18.982239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:18.980790 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:19.231681 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:19.231649 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" event={"ID":"d39b2299-705b-4969-8669-1d796017249d","Type":"ContainerStarted","Data":"0a22023c336bbefbe3445880df6ce7ec6015a472b64e572f4581831198b206e3"} Oct 30 06:07:19.244239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:19.235145 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:19.550387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:19.550282 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:07:19.555387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:19.550410 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 06:07:19.555676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:19.555635 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" podUID="d39b2299-705b-4969-8669-1d796017249d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 06:07:19.865636 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:19.865606 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:19.979376 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:19.979341 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:07:19.980684 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:19.980662 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:07:19.981349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:19.981327 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:07:20.458126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:20.458091 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:20.555256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:20.554544 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 06:07:20.555256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:20.554605 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" podUID="d39b2299-705b-4969-8669-1d796017249d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 06:07:20.630151 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:20.630099 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:20.772992 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:20.772910 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:21.032413 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:21.032319 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:21.220170 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:21.220140 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:21.391184 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:21.391158 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:21.559380 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:21.559347 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 06:07:21.559759 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:21.559409 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" podUID="d39b2299-705b-4969-8669-1d796017249d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 06:07:22.551062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:22.551029 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 06:07:22.551390 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:22.551361 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" podUID="d39b2299-705b-4969-8669-1d796017249d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 06:07:23.046335 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:23.046302 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:07:23.211663 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:23.211632 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:23.450575 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:23.450541 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:23.549679 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:23.549649 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Oct 30 06:07:23.549951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:23.549921 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" podUID="d39b2299-705b-4969-8669-1d796017249d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Oct 30 06:07:24.034072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:24.034043 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:24.181024 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:24.180993 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:24.340427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:24.340343 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:24.552587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:24.552556 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:07:24.553627 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:24.553603 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:07:25.151436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.151390 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:25.190182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.190151 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx"] Oct 30 06:07:25.195037 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.195012 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.242639 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.234581 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-aaf467\"/\"default-dockercfg-sjppv\"" Oct 30 06:07:25.242882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.237075 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-aaf467\"/\"kube-root-ca.crt\"" Oct 30 06:07:25.242882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.237242 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-aaf467\"/\"simulator-w2-aaf467-kserve-self-signed-certs\"" Oct 30 06:07:25.242882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.238866 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx"] Oct 30 06:07:25.243110 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.242746 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-aaf467\"/\"openshift-service-ca.crt\"" Oct 30 06:07:25.301330 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.301300 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-tmp-dir\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.301499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.301336 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-tls-certs\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.301499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.301440 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-model-cache\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.301593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.301499 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwvkf\" (UniqueName: \"kubernetes.io/projected/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-kube-api-access-qwvkf\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.301593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.301556 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-dshm\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.301593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.301584 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-home\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.326985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.326960 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:25.402529 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.402448 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-model-cache\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.402529 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.402519 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwvkf\" (UniqueName: \"kubernetes.io/projected/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-kube-api-access-qwvkf\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.402748 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.402572 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-dshm\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.402748 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.402608 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-home\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.402748 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.402674 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-tmp-dir\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.402748 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.402739 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-tls-certs\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.402945 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.402827 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-model-cache\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.403171 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.403145 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-tmp-dir\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.403171 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.403160 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-home\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.405154 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.405131 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-dshm\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.405739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.405718 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-tls-certs\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.426655 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.426629 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwvkf\" (UniqueName: \"kubernetes.io/projected/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-kube-api-access-qwvkf\") pod \"simulator-w2-aaf467-kserve-6d8f69c99b-55hbx\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.508576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.508547 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:25.813795 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:25.813762 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx"] Oct 30 06:07:25.816312 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:07:25.816282 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e6a2ea8_fe55_4e70_bf40_664ce2ba1ce3.slice/crio-9463bceb8812bf4bec64eaf27a75bb91d7bb4616e418540bf7aa8b8e9115cbf1 WatchSource:0}: Error finding container 9463bceb8812bf4bec64eaf27a75bb91d7bb4616e418540bf7aa8b8e9115cbf1: Status 404 returned error can't find the container with id 9463bceb8812bf4bec64eaf27a75bb91d7bb4616e418540bf7aa8b8e9115cbf1 Oct 30 06:07:26.302451 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:26.302419 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" event={"ID":"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3","Type":"ContainerStarted","Data":"9463bceb8812bf4bec64eaf27a75bb91d7bb4616e418540bf7aa8b8e9115cbf1"} Oct 30 06:07:27.258227 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:27.252194 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:28.318284 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:28.318245 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" event={"ID":"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3","Type":"ContainerStarted","Data":"405773f083aa8e5b74581b4f6ffd6f71a16028b93098ec9f1c3698b2786372dc"} Oct 30 06:07:28.363044 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:28.362993 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" podStartSLOduration=1.9023558010000001 podStartE2EDuration="3.36298031s" totalImagesPullingTime="1.460624509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:28.361579152 +0000 UTC m=+814.870599041" watchObservedRunningTime="2026-10-30 06:07:28.36298031 +0000 UTC m=+814.872000238" Oct 30 06:07:28.509520 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:28.509489 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:29.066709 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:29.066638 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:29.874134 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:29.874098 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp"] Oct 30 06:07:29.878151 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:29.878130 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp"] Oct 30 06:07:29.878293 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:29.878241 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:29.906443 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:29.906423 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-eaee88\"/\"openshift-service-ca.crt\"" Oct 30 06:07:29.906713 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:29.906681 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-eaee88\"/\"default-dockercfg-5fqvm\"" Oct 30 06:07:29.906713 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:29.906705 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-eaee88\"/\"kube-root-ca.crt\"" Oct 30 06:07:29.906840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:29.906764 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-eaee88\"/\"simulator-w4-eaee88-kserve-self-signed-certs\"" Oct 30 06:07:29.951723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:29.951694 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:29.971518 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:29.971478 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ggk8\" (UniqueName: \"kubernetes.io/projected/776861d0-f966-44ca-b0eb-22b1a1b47e79-kube-api-access-9ggk8\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:29.971682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:29.971533 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-home\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:29.971682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:29.971561 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-dshm\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:29.971682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:29.971595 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776861d0-f966-44ca-b0eb-22b1a1b47e79-tls-certs\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:29.971682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:29.971627 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-model-cache\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:29.971682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:29.971649 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-tmp-dir\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:30.072945 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:30.072902 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ggk8\" (UniqueName: \"kubernetes.io/projected/776861d0-f966-44ca-b0eb-22b1a1b47e79-kube-api-access-9ggk8\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:30.073146 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:30.072960 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-home\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:30.073146 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:30.072994 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-dshm\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:30.073146 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:30.073035 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776861d0-f966-44ca-b0eb-22b1a1b47e79-tls-certs\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:30.073146 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:30.073067 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-model-cache\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:30.073146 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:30.073094 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-tmp-dir\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:30.073487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:30.073469 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-tmp-dir\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:30.073899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:30.073882 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-home\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:30.074333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:30.074316 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-model-cache\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:30.076424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:30.076403 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-dshm\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:30.076836 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:30.076819 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776861d0-f966-44ca-b0eb-22b1a1b47e79-tls-certs\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:30.097896 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:30.097865 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ggk8\" (UniqueName: \"kubernetes.io/projected/776861d0-f966-44ca-b0eb-22b1a1b47e79-kube-api-access-9ggk8\") pod \"simulator-w4-eaee88-kserve-67c4785f4c-gn4zp\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:30.194673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:30.193432 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:30.579613 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:30.579505 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp"] Oct 30 06:07:30.583023 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:07:30.582994 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod776861d0_f966_44ca_b0eb_22b1a1b47e79.slice/crio-2a7229960c9d9e03d4af030b11d6a9a61ee3380881cac174849514e06850ee9c WatchSource:0}: Error finding container 2a7229960c9d9e03d4af030b11d6a9a61ee3380881cac174849514e06850ee9c: Status 404 returned error can't find the container with id 2a7229960c9d9e03d4af030b11d6a9a61ee3380881cac174849514e06850ee9c Oct 30 06:07:31.022036 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:31.021992 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:31.359371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:31.359340 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" event={"ID":"776861d0-f966-44ca-b0eb-22b1a1b47e79","Type":"ContainerStarted","Data":"484d8bed666746ba0dca30291dc74239b714663a8d0df6943654b6f58390a63d"} Oct 30 06:07:31.359569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:31.359543 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" event={"ID":"776861d0-f966-44ca-b0eb-22b1a1b47e79","Type":"ContainerStarted","Data":"2a7229960c9d9e03d4af030b11d6a9a61ee3380881cac174849514e06850ee9c"} Oct 30 06:07:31.404671 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:31.404620 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" podStartSLOduration=2.404604297 podStartE2EDuration="2.404604297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:31.402514276 +0000 UTC m=+817.911534162" watchObservedRunningTime="2026-10-30 06:07:31.404604297 +0000 UTC m=+817.913624188" Oct 30 06:07:32.195729 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:32.195697 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:32.851788 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:32.851748 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:34.142936 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:07:34.142857 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:07:34.777932 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:34.777905 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:35.383442 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:35.383409 6999 generic.go:356] "Generic (PLEG): container finished" podID="0ae13585-2214-4d4c-87ec-0e7ee0b57213" containerID="0d7af5f97642789db4d0ead75e21e35538038446d20d52c0654a80161e93cbae" exitCode=0 Oct 30 06:07:35.383442 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:35.383446 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" event={"ID":"0ae13585-2214-4d4c-87ec-0e7ee0b57213","Type":"ContainerDied","Data":"0d7af5f97642789db4d0ead75e21e35538038446d20d52c0654a80161e93cbae"} Oct 30 06:07:35.383870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:35.383759 6999 scope.go:122] "RemoveContainer" containerID="0d7af5f97642789db4d0ead75e21e35538038446d20d52c0654a80161e93cbae" Oct 30 06:07:35.385317 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:35.385275 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:35.385436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:35.385365 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:35.385436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:35.385405 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:35.511141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:35.508968 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:35.522600 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:35.522574 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:35.966046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:35.966010 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:36.105958 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.105927 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm"] Oct 30 06:07:36.110002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.109978 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.139536 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.139507 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a519e9\"/\"kube-root-ca.crt\"" Oct 30 06:07:36.150303 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.139865 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a519e9\"/\"default-dockercfg-9mttb\"" Oct 30 06:07:36.150587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.139910 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a519e9\"/\"simulator-w0-a519e9-kserve-self-signed-certs\"" Oct 30 06:07:36.150692 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.142588 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm"] Oct 30 06:07:36.150811 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.142678 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a519e9\"/\"openshift-service-ca.crt\"" Oct 30 06:07:36.211121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.211057 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:36.268295 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.263808 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-home\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.268295 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.263861 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-model-cache\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.268295 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.263944 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-dshm\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.268295 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.263998 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-tmp-dir\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.268295 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.264033 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg66v\" (UniqueName: \"kubernetes.io/projected/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-kube-api-access-rg66v\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.268295 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.264105 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-tls-certs\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.365237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.364878 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-tmp-dir\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.365237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.364951 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg66v\" (UniqueName: \"kubernetes.io/projected/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-kube-api-access-rg66v\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.365237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.365060 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-tls-certs\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.365237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.365158 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-home\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.365237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.365183 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-model-cache\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.365677 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.365289 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-dshm\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.366525 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.366474 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-home\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.366786 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.366763 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-model-cache\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.367055 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.367028 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-tmp-dir\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.368261 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.368200 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-dshm\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.369050 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.369028 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-tls-certs\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.390755 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.390729 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg66v\" (UniqueName: \"kubernetes.io/projected/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-kube-api-access-rg66v\") pod \"simulator-w0-a519e9-kserve-5898f479c-8wxtm\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.394493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.394465 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" event={"ID":"0ae13585-2214-4d4c-87ec-0e7ee0b57213","Type":"ContainerStarted","Data":"c82fb6ce0e1eb06c8cb7bcb05e21937c8e8f5b9dbf2fc888cecc99938b903f05"} Oct 30 06:07:36.406907 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.406878 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:07:36.422835 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.422814 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:36.659703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.659648 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:36.764260 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.764198 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm"] Oct 30 06:07:36.767947 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:07:36.767920 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79b8b3f2_6e6b_49cb_a973_8fb098ff3bc4.slice/crio-d42985b51bb9386f7079f03ef5a715db6a107bf8a22f613f3e00287ca1f69404 WatchSource:0}: Error finding container d42985b51bb9386f7079f03ef5a715db6a107bf8a22f613f3e00287ca1f69404: Status 404 returned error can't find the container with id d42985b51bb9386f7079f03ef5a715db6a107bf8a22f613f3e00287ca1f69404 Oct 30 06:07:36.823748 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.823718 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:36.973616 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:36.973529 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:37.387745 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:37.387714 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:37.388904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:37.388879 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 06:07:37.389033 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:37.388934 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" podUID="0ae13585-2214-4d4c-87ec-0e7ee0b57213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 06:07:37.406487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:37.406456 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" event={"ID":"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4","Type":"ContainerStarted","Data":"c8a4a9702019f7ae10e7e131e4faf9f827811d093eb45e43b672d37e2a5c9f84"} Oct 30 06:07:37.406487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:37.406491 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" event={"ID":"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4","Type":"ContainerStarted","Data":"d42985b51bb9386f7079f03ef5a715db6a107bf8a22f613f3e00287ca1f69404"} Oct 30 06:07:37.423621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:37.423596 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:38.388653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:38.388612 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 06:07:38.388840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:38.388681 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" podUID="0ae13585-2214-4d4c-87ec-0e7ee0b57213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 06:07:38.998001 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:38.997965 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:39.140240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:39.138357 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:39.388479 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:39.388446 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 06:07:39.388664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:39.388496 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" podUID="0ae13585-2214-4d4c-87ec-0e7ee0b57213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 06:07:40.196045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:40.196009 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:40.211714 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:40.211640 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:40.259243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:40.258204 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" podStartSLOduration=4.258186062 podStartE2EDuration="4.258186062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:37.460536992 +0000 UTC m=+823.969556906" watchObservedRunningTime="2026-10-30 06:07:40.258186062 +0000 UTC m=+826.767205951" Oct 30 06:07:40.388369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:40.388326 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 30 06:07:40.388566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:40.388386 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" podUID="0ae13585-2214-4d4c-87ec-0e7ee0b57213" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 30 06:07:40.467348 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:40.467271 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:07:41.066238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:41.066067 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:41.457750 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:41.457718 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:41.460276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:41.460254 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:07:42.827934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:42.827898 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:44.282951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:44.282913 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:44.633569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:44.633539 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:46.423024 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:46.422991 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:46.435477 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:46.435427 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:46.435628 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:46.435596 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:46.482490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:46.482461 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:07:48.245008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:48.244975 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:50.095364 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:50.095329 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:50.242664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:50.242622 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:50.592061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:50.592012 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:51.928147 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:51.928095 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:53.718446 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:53.718407 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:55.537468 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:55.537417 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:57.227120 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:57.227082 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:57.372327 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:57.372244 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:57.879852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:57.879806 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:58.838311 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:58.838281 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:59.226344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:59.226311 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:07:59.878483 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:07:59.878434 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:01.144746 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:01.144713 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:02.932142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:02.932079 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:04.143587 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:08:04.143547 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:08:04.638132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:04.638079 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:06.218578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:06.218540 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:06.374258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:06.372041 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:08.226092 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:08.226054 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:10.105475 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:10.105443 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:11.517158 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.517130 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm"] Oct 30 06:08:11.520575 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.520558 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.554404 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.554372 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-eaee88\"/\"premium-w4-eaee88-kserve-self-signed-certs\"" Oct 30 06:08:11.555843 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.555818 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm"] Oct 30 06:08:11.634240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.634191 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-dshm\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.634422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.634259 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd2wh\" (UniqueName: \"kubernetes.io/projected/b2701c51-6377-411c-9924-715ff0765564-kube-api-access-jd2wh\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.634422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.634379 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-model-cache\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.634554 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.634435 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-tmp-dir\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.634606 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.634555 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-home\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.634672 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.634651 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2701c51-6377-411c-9924-715ff0765564-tls-certs\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.642191 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.642162 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:11.735891 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.735819 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2701c51-6377-411c-9924-715ff0765564-tls-certs\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.736083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.735942 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-dshm\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.736083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.735980 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd2wh\" (UniqueName: \"kubernetes.io/projected/b2701c51-6377-411c-9924-715ff0765564-kube-api-access-jd2wh\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.736083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.736071 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-model-cache\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.736188 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.736101 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-tmp-dir\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.736188 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.736154 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-home\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.736778 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.736753 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-model-cache\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.739028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.736840 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-home\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.739306 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.736932 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-tmp-dir\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.739463 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.739443 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-dshm\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.739977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.739957 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2701c51-6377-411c-9924-715ff0765564-tls-certs\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.761623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.761598 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd2wh\" (UniqueName: \"kubernetes.io/projected/b2701c51-6377-411c-9924-715ff0765564-kube-api-access-jd2wh\") pod \"premium-w4-eaee88-kserve-5c444cbcf4-dj7dm\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.834512 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.834450 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:11.991588 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:11.991559 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:12.163306 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:12.163260 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm"] Oct 30 06:08:12.165973 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:08:12.165951 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2701c51_6377_411c_9924_715ff0765564.slice/crio-c19d7e80656cbd1366f01ce47e41760296daf58f197c9dfa44ff38afa7d2f250 WatchSource:0}: Error finding container c19d7e80656cbd1366f01ce47e41760296daf58f197c9dfa44ff38afa7d2f250: Status 404 returned error can't find the container with id c19d7e80656cbd1366f01ce47e41760296daf58f197c9dfa44ff38afa7d2f250 Oct 30 06:08:12.266364 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:12.266333 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:12.613666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:12.613633 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" event={"ID":"b2701c51-6377-411c-9924-715ff0765564","Type":"ContainerStarted","Data":"7f41ba26531816fdba6acf988b99d9b60481a2c5173f25fa42f473cb043b6102"} Oct 30 06:08:12.614089 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:12.613673 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" event={"ID":"b2701c51-6377-411c-9924-715ff0765564","Type":"ContainerStarted","Data":"c19d7e80656cbd1366f01ce47e41760296daf58f197c9dfa44ff38afa7d2f250"} Oct 30 06:08:12.656564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:12.656519 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" podStartSLOduration=1.656506679 podStartE2EDuration="1.656506679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:12.654745509 +0000 UTC m=+859.163765400" watchObservedRunningTime="2026-10-30 06:08:12.656506679 +0000 UTC m=+859.165526566" Oct 30 06:08:12.835581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:12.835540 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:14.055569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:14.055525 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:15.756430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:15.756393 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:17.608375 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:17.608342 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:19.593406 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:19.593375 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:21.469884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:21.469851 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:21.835334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:21.835241 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:21.848052 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:21.848025 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:22.675512 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:22.675488 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:08:23.392995 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:23.392967 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:25.331844 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:25.331811 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:26.725378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:26.725344 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:27.176685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:27.176651 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:28.923739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:28.923710 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:30.643760 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:30.643722 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:32.421007 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:32.420966 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:34.150752 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:08:34.150710 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:08:34.361129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.361066 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h"] Oct 30 06:08:34.368138 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.368114 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.369705 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.369555 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:34.383668 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.383633 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-aaf467-openshift-default-dockercfg-txgqd\"" Oct 30 06:08:34.396904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.396878 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h"] Oct 30 06:08:34.464691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.464604 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6aefe60-58f8-490e-9c3d-e062edf6e038-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.464691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.464651 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8cd\" (UniqueName: \"kubernetes.io/projected/a6aefe60-58f8-490e-9c3d-e062edf6e038-kube-api-access-jq8cd\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.464921 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.464769 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.464921 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.464829 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.464921 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.464881 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.465104 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.464996 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.465104 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.465067 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.465246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.465101 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.465246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.465130 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.566527 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.566497 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.566527 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.566531 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.566757 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.566548 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.566757 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.566610 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6aefe60-58f8-490e-9c3d-e062edf6e038-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.566757 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.566634 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8cd\" (UniqueName: \"kubernetes.io/projected/a6aefe60-58f8-490e-9c3d-e062edf6e038-kube-api-access-jq8cd\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.566951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.566772 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.566951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.566824 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.566951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.566874 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.566951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.566920 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.567312 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.567284 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.567694 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.567672 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.567852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.567713 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.567965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.567945 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6aefe60-58f8-490e-9c3d-e062edf6e038-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.568068 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.568046 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.569760 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.569734 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.570014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.569994 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.603374 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.603351 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.603507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.603490 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8cd\" (UniqueName: \"kubernetes.io/projected/a6aefe60-58f8-490e-9c3d-e062edf6e038-kube-api-access-jq8cd\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:34.685552 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:34.685528 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:35.005176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:35.005119 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h"] Oct 30 06:08:35.007570 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:08:35.007544 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6aefe60_58f8_490e_9c3d_e062edf6e038.slice/crio-b262ddd98600a274b2e7bfc1a5d9914e4a8b5dbbbc1cd4c98545f327cc43dcc0 WatchSource:0}: Error finding container b262ddd98600a274b2e7bfc1a5d9914e4a8b5dbbbc1cd4c98545f327cc43dcc0: Status 404 returned error can't find the container with id b262ddd98600a274b2e7bfc1a5d9914e4a8b5dbbbc1cd4c98545f327cc43dcc0 Oct 30 06:08:35.009822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:35.009789 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:08:35.009916 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:35.009853 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:08:35.009916 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:35.009881 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:08:35.770384 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:35.770347 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" event={"ID":"a6aefe60-58f8-490e-9c3d-e062edf6e038","Type":"ContainerStarted","Data":"4f4e9763b521bf0750a5af508be168427035d62e36b09536a0ca1bc024e38c8a"} Oct 30 06:08:35.770783 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:35.770398 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" event={"ID":"a6aefe60-58f8-490e-9c3d-e062edf6e038","Type":"ContainerStarted","Data":"b262ddd98600a274b2e7bfc1a5d9914e4a8b5dbbbc1cd4c98545f327cc43dcc0"} Oct 30 06:08:35.854008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:35.853938 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" podStartSLOduration=1.853915583 podStartE2EDuration="1.853915583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:35.845709029 +0000 UTC m=+882.354728915" watchObservedRunningTime="2026-10-30 06:08:35.853915583 +0000 UTC m=+882.362935470" Oct 30 06:08:36.686080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:36.686050 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:36.687619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:36.687592 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 06:08:36.687742 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:36.687639 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" podUID="a6aefe60-58f8-490e-9c3d-e062edf6e038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 06:08:36.762000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:36.761950 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:37.686625 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:37.686591 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 06:08:37.687133 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:37.686644 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" podUID="a6aefe60-58f8-490e-9c3d-e062edf6e038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 06:08:38.686596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:38.686557 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 06:08:38.686814 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:38.686626 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" podUID="a6aefe60-58f8-490e-9c3d-e062edf6e038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 06:08:38.715944 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:38.715908 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:39.686736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:39.686701 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 06:08:39.686929 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:39.686756 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" podUID="a6aefe60-58f8-490e-9c3d-e062edf6e038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 06:08:40.683948 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:40.683921 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:40.686035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:40.686010 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 06:08:40.686169 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:40.686052 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" podUID="a6aefe60-58f8-490e-9c3d-e062edf6e038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 06:08:41.691017 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:41.690986 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:41.691442 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:41.691336 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:41.691941 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:41.691920 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:08:42.776518 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:42.776449 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:44.621595 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:44.621561 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:46.597391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:46.597360 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:48.360474 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:48.360439 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:50.142451 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:50.142416 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:52.127476 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:52.127447 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:54.026611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:54.026578 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:56.036824 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:56.036789 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:57.987635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:57.987583 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:58.139315 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.139286 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:58.688009 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.687976 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt"] Oct 30 06:08:58.692853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.692827 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" Oct 30 06:08:58.711153 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.711131 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt"] Oct 30 06:08:58.720691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.720669 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:08:58.720818 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.720729 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:08:58.734204 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.734179 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj2qz\" (UniqueName: \"kubernetes.io/projected/77704d62-3fd8-428d-b998-cceb2d742116-kube-api-access-mj2qz\") pod \"maas-api-revoke-sub-8fc9fd2f-fjwmt\" (UID: \"77704d62-3fd8-428d-b998-cceb2d742116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" Oct 30 06:08:58.734353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.734299 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77704d62-3fd8-428d-b998-cceb2d742116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fc9fd2f-fjwmt\" (UID: \"77704d62-3fd8-428d-b998-cceb2d742116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" Oct 30 06:08:58.734432 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.734412 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77704d62-3fd8-428d-b998-cceb2d742116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fc9fd2f-fjwmt\" (UID: \"77704d62-3fd8-428d-b998-cceb2d742116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" Oct 30 06:08:58.835173 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.835143 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj2qz\" (UniqueName: \"kubernetes.io/projected/77704d62-3fd8-428d-b998-cceb2d742116-kube-api-access-mj2qz\") pod \"maas-api-revoke-sub-8fc9fd2f-fjwmt\" (UID: \"77704d62-3fd8-428d-b998-cceb2d742116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" Oct 30 06:08:58.835333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.835205 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77704d62-3fd8-428d-b998-cceb2d742116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fc9fd2f-fjwmt\" (UID: \"77704d62-3fd8-428d-b998-cceb2d742116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" Oct 30 06:08:58.835333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.835251 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77704d62-3fd8-428d-b998-cceb2d742116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fc9fd2f-fjwmt\" (UID: \"77704d62-3fd8-428d-b998-cceb2d742116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" Oct 30 06:08:58.835816 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.835794 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77704d62-3fd8-428d-b998-cceb2d742116-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fc9fd2f-fjwmt\" (UID: \"77704d62-3fd8-428d-b998-cceb2d742116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" Oct 30 06:08:58.837863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.837834 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77704d62-3fd8-428d-b998-cceb2d742116-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fc9fd2f-fjwmt\" (UID: \"77704d62-3fd8-428d-b998-cceb2d742116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" Oct 30 06:08:58.858110 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.858086 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj2qz\" (UniqueName: \"kubernetes.io/projected/77704d62-3fd8-428d-b998-cceb2d742116-kube-api-access-mj2qz\") pod \"maas-api-revoke-sub-8fc9fd2f-fjwmt\" (UID: \"77704d62-3fd8-428d-b998-cceb2d742116\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" Oct 30 06:08:58.952862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:58.952804 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:59.006368 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:59.006345 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" Oct 30 06:08:59.310946 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:59.310923 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt"] Oct 30 06:08:59.312846 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:08:59.312813 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77704d62_3fd8_428d_b998_cceb2d742116.slice/crio-7044d111876fdf6a5cf8b9c2242b6484fd1d491c7f074cf97dadf895ee6e602f WatchSource:0}: Error finding container 7044d111876fdf6a5cf8b9c2242b6484fd1d491c7f074cf97dadf895ee6e602f: Status 404 returned error can't find the container with id 7044d111876fdf6a5cf8b9c2242b6484fd1d491c7f074cf97dadf895ee6e602f Oct 30 06:08:59.860344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:59.860315 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:08:59.899311 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:59.899270 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" event={"ID":"77704d62-3fd8-428d-b998-cceb2d742116","Type":"ContainerStarted","Data":"4485ee3130064931e516b8601c11574f8bf762c7b1eab9fb9c52ad59f16c9aaf"} Oct 30 06:08:59.899455 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:59.899331 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" event={"ID":"77704d62-3fd8-428d-b998-cceb2d742116","Type":"ContainerStarted","Data":"7044d111876fdf6a5cf8b9c2242b6484fd1d491c7f074cf97dadf895ee6e602f"} Oct 30 06:08:59.946533 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:08:59.946491 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" podStartSLOduration=1.946476018 podStartE2EDuration="1.946476018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:08:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:59.94475157 +0000 UTC m=+906.453771458" watchObservedRunningTime="2026-10-30 06:08:59.946476018 +0000 UTC m=+906.455495905" Oct 30 06:09:00.039543 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:00.039507 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:00.925815 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:00.925776 6999 generic.go:356] "Generic (PLEG): container finished" podID="77704d62-3fd8-428d-b998-cceb2d742116" containerID="4485ee3130064931e516b8601c11574f8bf762c7b1eab9fb9c52ad59f16c9aaf" exitCode=0 Oct 30 06:09:00.926024 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:00.925858 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" event={"ID":"77704d62-3fd8-428d-b998-cceb2d742116","Type":"ContainerDied","Data":"4485ee3130064931e516b8601c11574f8bf762c7b1eab9fb9c52ad59f16c9aaf"} Oct 30 06:09:01.697073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:01.697025 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:02.072852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:02.072826 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" Oct 30 06:09:02.180779 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:02.180750 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77704d62-3fd8-428d-b998-cceb2d742116-kube-api-access-mj2qz\" (UniqueName: \"kubernetes.io/projected/77704d62-3fd8-428d-b998-cceb2d742116-kube-api-access-mj2qz\") pod \"77704d62-3fd8-428d-b998-cceb2d742116\" (UID: \"77704d62-3fd8-428d-b998-cceb2d742116\") " Oct 30 06:09:02.180943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:02.180787 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77704d62-3fd8-428d-b998-cceb2d742116-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77704d62-3fd8-428d-b998-cceb2d742116-maas-api-openshift-ca\") pod \"77704d62-3fd8-428d-b998-cceb2d742116\" (UID: \"77704d62-3fd8-428d-b998-cceb2d742116\") " Oct 30 06:09:02.181030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:02.181013 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77704d62-3fd8-428d-b998-cceb2d742116-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77704d62-3fd8-428d-b998-cceb2d742116-maas-api-xks-ca\") pod \"77704d62-3fd8-428d-b998-cceb2d742116\" (UID: \"77704d62-3fd8-428d-b998-cceb2d742116\") " Oct 30 06:09:02.181252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:02.181201 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77704d62-3fd8-428d-b998-cceb2d742116-maas-api-openshift-ca" pod "77704d62-3fd8-428d-b998-cceb2d742116" (UID: "77704d62-3fd8-428d-b998-cceb2d742116"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:02.181459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:02.181439 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77704d62-3fd8-428d-b998-cceb2d742116-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:09:02.183043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:02.183023 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77704d62-3fd8-428d-b998-cceb2d742116-kube-api-access-mj2qz" pod "77704d62-3fd8-428d-b998-cceb2d742116" (UID: "77704d62-3fd8-428d-b998-cceb2d742116"). InnerVolumeSpecName "kube-api-access-mj2qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:02.183144 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:02.183080 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77704d62-3fd8-428d-b998-cceb2d742116-maas-api-xks-ca" pod "77704d62-3fd8-428d-b998-cceb2d742116" (UID: "77704d62-3fd8-428d-b998-cceb2d742116"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:02.282979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:02.282891 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77704d62-3fd8-428d-b998-cceb2d742116-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:09:02.282979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:02.282923 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj2qz\" (UniqueName: \"kubernetes.io/projected/77704d62-3fd8-428d-b998-cceb2d742116-kube-api-access-mj2qz\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:09:02.937042 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:02.937013 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" Oct 30 06:09:02.937472 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:02.937076 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt" event={"ID":"77704d62-3fd8-428d-b998-cceb2d742116","Type":"ContainerDied","Data":"7044d111876fdf6a5cf8b9c2242b6484fd1d491c7f074cf97dadf895ee6e602f"} Oct 30 06:09:02.937472 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:02.937115 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7044d111876fdf6a5cf8b9c2242b6484fd1d491c7f074cf97dadf895ee6e602f" Oct 30 06:09:03.432462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:03.432381 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:04.141229 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:09:04.141176 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:09:05.291004 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:05.290957 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:07.030489 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:07.030432 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:08.942017 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:08.941986 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:10.856780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:10.856746 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:12.736227 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:12.736190 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:14.531669 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:14.531622 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:16.103183 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.103153 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh"] Oct 30 06:09:16.103702 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.103688 6999 cpu_manager.go:414] "containerMap: removing container" podUID="77704d62-3fd8-428d-b998-cceb2d742116" containerName="revoke-keys" Oct 30 06:09:16.103766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.103705 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77704d62-3fd8-428d-b998-cceb2d742116" containerName="revoke-keys" Oct 30 06:09:16.103830 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.103820 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="77704d62-3fd8-428d-b998-cceb2d742116" containerName="revoke-keys" Oct 30 06:09:16.107175 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.107155 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" Oct 30 06:09:16.121773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.121752 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:09:16.133488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.133468 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:09:16.135206 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.135179 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh"] Oct 30 06:09:16.250989 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.250959 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0966f3b6-5625-4967-a6b7-64289ea9f9b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ca1629b-dn5dh\" (UID: \"0966f3b6-5625-4967-a6b7-64289ea9f9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" Oct 30 06:09:16.251178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.251023 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d4vl\" (UniqueName: \"kubernetes.io/projected/0966f3b6-5625-4967-a6b7-64289ea9f9b8-kube-api-access-4d4vl\") pod \"maas-api-revoke-sub-4ca1629b-dn5dh\" (UID: \"0966f3b6-5625-4967-a6b7-64289ea9f9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" Oct 30 06:09:16.251178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.251121 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0966f3b6-5625-4967-a6b7-64289ea9f9b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ca1629b-dn5dh\" (UID: \"0966f3b6-5625-4967-a6b7-64289ea9f9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" Oct 30 06:09:16.311323 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.311294 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:16.352866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.352827 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d4vl\" (UniqueName: \"kubernetes.io/projected/0966f3b6-5625-4967-a6b7-64289ea9f9b8-kube-api-access-4d4vl\") pod \"maas-api-revoke-sub-4ca1629b-dn5dh\" (UID: \"0966f3b6-5625-4967-a6b7-64289ea9f9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" Oct 30 06:09:16.353036 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.352912 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0966f3b6-5625-4967-a6b7-64289ea9f9b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ca1629b-dn5dh\" (UID: \"0966f3b6-5625-4967-a6b7-64289ea9f9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" Oct 30 06:09:16.353113 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.353060 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0966f3b6-5625-4967-a6b7-64289ea9f9b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ca1629b-dn5dh\" (UID: \"0966f3b6-5625-4967-a6b7-64289ea9f9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" Oct 30 06:09:16.353884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.353830 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0966f3b6-5625-4967-a6b7-64289ea9f9b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ca1629b-dn5dh\" (UID: \"0966f3b6-5625-4967-a6b7-64289ea9f9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" Oct 30 06:09:16.355468 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.355447 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0966f3b6-5625-4967-a6b7-64289ea9f9b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ca1629b-dn5dh\" (UID: \"0966f3b6-5625-4967-a6b7-64289ea9f9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" Oct 30 06:09:16.377736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.377713 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d4vl\" (UniqueName: \"kubernetes.io/projected/0966f3b6-5625-4967-a6b7-64289ea9f9b8-kube-api-access-4d4vl\") pod \"maas-api-revoke-sub-4ca1629b-dn5dh\" (UID: \"0966f3b6-5625-4967-a6b7-64289ea9f9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" Oct 30 06:09:16.420662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.420631 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" Oct 30 06:09:16.721070 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:16.721039 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh"] Oct 30 06:09:16.724137 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:09:16.724107 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0966f3b6_5625_4967_a6b7_64289ea9f9b8.slice/crio-880536ce96ede43e822115d2642af7b02b9e8fc7f94682a7050a96bb33bc0ad3 WatchSource:0}: Error finding container 880536ce96ede43e822115d2642af7b02b9e8fc7f94682a7050a96bb33bc0ad3: Status 404 returned error can't find the container with id 880536ce96ede43e822115d2642af7b02b9e8fc7f94682a7050a96bb33bc0ad3 Oct 30 06:09:17.038501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:17.038473 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" event={"ID":"0966f3b6-5625-4967-a6b7-64289ea9f9b8","Type":"ContainerStarted","Data":"880536ce96ede43e822115d2642af7b02b9e8fc7f94682a7050a96bb33bc0ad3"} Oct 30 06:09:18.050677 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:18.050637 6999 generic.go:356] "Generic (PLEG): container finished" podID="0966f3b6-5625-4967-a6b7-64289ea9f9b8" containerID="32888dd8882c6ba67a05fc16d14821345c66a06ca184506b69e299196a1840e2" exitCode=0 Oct 30 06:09:18.051257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:18.050756 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" event={"ID":"0966f3b6-5625-4967-a6b7-64289ea9f9b8","Type":"ContainerDied","Data":"32888dd8882c6ba67a05fc16d14821345c66a06ca184506b69e299196a1840e2"} Oct 30 06:09:18.209859 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:18.209824 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:18.713592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:18.713554 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:19.204203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:19.204179 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" Oct 30 06:09:19.280539 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:19.280510 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0966f3b6-5625-4967-a6b7-64289ea9f9b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0966f3b6-5625-4967-a6b7-64289ea9f9b8-maas-api-openshift-ca\") pod \"0966f3b6-5625-4967-a6b7-64289ea9f9b8\" (UID: \"0966f3b6-5625-4967-a6b7-64289ea9f9b8\") " Oct 30 06:09:19.280720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:19.280570 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0966f3b6-5625-4967-a6b7-64289ea9f9b8-kube-api-access-4d4vl\" (UniqueName: \"kubernetes.io/projected/0966f3b6-5625-4967-a6b7-64289ea9f9b8-kube-api-access-4d4vl\") pod \"0966f3b6-5625-4967-a6b7-64289ea9f9b8\" (UID: \"0966f3b6-5625-4967-a6b7-64289ea9f9b8\") " Oct 30 06:09:19.280720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:19.280621 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0966f3b6-5625-4967-a6b7-64289ea9f9b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0966f3b6-5625-4967-a6b7-64289ea9f9b8-maas-api-xks-ca\") pod \"0966f3b6-5625-4967-a6b7-64289ea9f9b8\" (UID: \"0966f3b6-5625-4967-a6b7-64289ea9f9b8\") " Oct 30 06:09:19.280934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:19.280912 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0966f3b6-5625-4967-a6b7-64289ea9f9b8-maas-api-openshift-ca" pod "0966f3b6-5625-4967-a6b7-64289ea9f9b8" (UID: "0966f3b6-5625-4967-a6b7-64289ea9f9b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:19.281095 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:19.281079 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0966f3b6-5625-4967-a6b7-64289ea9f9b8-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.282695 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:19.282662 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0966f3b6-5625-4967-a6b7-64289ea9f9b8-kube-api-access-4d4vl" pod "0966f3b6-5625-4967-a6b7-64289ea9f9b8" (UID: "0966f3b6-5625-4967-a6b7-64289ea9f9b8"). InnerVolumeSpecName "kube-api-access-4d4vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:19.283017 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:19.282997 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0966f3b6-5625-4967-a6b7-64289ea9f9b8-maas-api-xks-ca" pod "0966f3b6-5625-4967-a6b7-64289ea9f9b8" (UID: "0966f3b6-5625-4967-a6b7-64289ea9f9b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:19.382230 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:19.382179 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d4vl\" (UniqueName: \"kubernetes.io/projected/0966f3b6-5625-4967-a6b7-64289ea9f9b8-kube-api-access-4d4vl\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.382230 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:19.382222 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0966f3b6-5625-4967-a6b7-64289ea9f9b8-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:09:20.039994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:20.039964 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:20.066271 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:20.066233 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" event={"ID":"0966f3b6-5625-4967-a6b7-64289ea9f9b8","Type":"ContainerDied","Data":"880536ce96ede43e822115d2642af7b02b9e8fc7f94682a7050a96bb33bc0ad3"} Oct 30 06:09:20.066527 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:20.066299 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="880536ce96ede43e822115d2642af7b02b9e8fc7f94682a7050a96bb33bc0ad3" Oct 30 06:09:20.066527 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:20.066384 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh" Oct 30 06:09:21.829153 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:21.829117 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:23.650409 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:23.650359 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:25.535665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:25.535636 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:27.427166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:27.427135 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:29.240708 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:29.240674 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:31.230646 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:31.230607 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:33.023305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:33.023267 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:34.143833 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:09:34.143790 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:09:34.829934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:34.829885 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:36.069871 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.069831 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht"] Oct 30 06:09:36.070508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.070489 6999 cpu_manager.go:414] "containerMap: removing container" podUID="0966f3b6-5625-4967-a6b7-64289ea9f9b8" containerName="revoke-keys" Oct 30 06:09:36.070599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.070509 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0966f3b6-5625-4967-a6b7-64289ea9f9b8" containerName="revoke-keys" Oct 30 06:09:36.070653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.070635 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="0966f3b6-5625-4967-a6b7-64289ea9f9b8" containerName="revoke-keys" Oct 30 06:09:36.073725 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.073707 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.093223 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.093184 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht"] Oct 30 06:09:36.102610 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.102584 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8d6349\"/\"kube-root-ca.crt\"" Oct 30 06:09:36.102757 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.102673 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8d6349\"/\"distinct-w3-8d6349-kserve-self-signed-certs\"" Oct 30 06:09:36.102757 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.102712 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8d6349\"/\"default-dockercfg-xkpcw\"" Oct 30 06:09:36.102757 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.102742 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8d6349\"/\"openshift-service-ca.crt\"" Oct 30 06:09:36.161099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.161058 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:36.170498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.170463 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-home\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.170660 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.170549 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-model-cache\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.170660 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.170620 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987180e9-4669-4740-b2df-effebab90159-tls-certs\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.170660 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.170644 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-tmp-dir\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.170844 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.170697 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-dshm\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.170928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.170902 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f626l\" (UniqueName: \"kubernetes.io/projected/987180e9-4669-4740-b2df-effebab90159-kube-api-access-f626l\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.271831 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.271797 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f626l\" (UniqueName: \"kubernetes.io/projected/987180e9-4669-4740-b2df-effebab90159-kube-api-access-f626l\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.272054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.271871 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-home\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.272054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.271931 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-model-cache\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.272054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.271958 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987180e9-4669-4740-b2df-effebab90159-tls-certs\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.272054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.271983 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-tmp-dir\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.272054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.272032 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-dshm\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.272399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.272364 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-home\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.272541 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.272398 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-model-cache\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.272628 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.272611 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-tmp-dir\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.277465 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.277443 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-dshm\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.289866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.289798 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987180e9-4669-4740-b2df-effebab90159-tls-certs\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.307119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.307079 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f626l\" (UniqueName: \"kubernetes.io/projected/987180e9-4669-4740-b2df-effebab90159-kube-api-access-f626l\") pod \"distinct-w3-8d6349-kserve-657d68664b-hrwht\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.387088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.387053 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:36.692554 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.692534 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht"] Oct 30 06:09:36.694931 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:09:36.694902 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod987180e9_4669_4740_b2df_effebab90159.slice/crio-8d621f4d7cea4e7bc584bfaca3b4cff1e385f7d6317146ec2b369f91b1f6f04c WatchSource:0}: Error finding container 8d621f4d7cea4e7bc584bfaca3b4cff1e385f7d6317146ec2b369f91b1f6f04c: Status 404 returned error can't find the container with id 8d621f4d7cea4e7bc584bfaca3b4cff1e385f7d6317146ec2b369f91b1f6f04c Oct 30 06:09:36.755696 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:36.755646 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:37.162590 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:37.162555 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" event={"ID":"987180e9-4669-4740-b2df-effebab90159","Type":"ContainerStarted","Data":"0869967d3ebe554007b96692fc67efc53ab48de3cebf42b267fde945e67b8b2c"} Oct 30 06:09:37.162590 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:37.162602 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" event={"ID":"987180e9-4669-4740-b2df-effebab90159","Type":"ContainerStarted","Data":"8d621f4d7cea4e7bc584bfaca3b4cff1e385f7d6317146ec2b369f91b1f6f04c"} Oct 30 06:09:37.204593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:37.204546 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" podStartSLOduration=1.204532162 podStartE2EDuration="1.204532162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:37.202407892 +0000 UTC m=+943.711427779" watchObservedRunningTime="2026-10-30 06:09:37.204532162 +0000 UTC m=+943.713552048" Oct 30 06:09:37.387465 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:37.387433 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:38.548181 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:38.548105 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:40.432848 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:40.432817 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:41.788085 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:41.788047 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb"] Oct 30 06:09:41.797087 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:41.797064 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:41.809315 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:41.809281 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb"] Oct 30 06:09:41.818575 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:41.818555 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-aaf467\"/\"distinct-w2-aaf467-kserve-self-signed-certs\"" Oct 30 06:09:41.864379 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:41.864335 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:41.933506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:41.933469 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-home\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:41.933506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:41.933509 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-dshm\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:41.933775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:41.933752 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-model-cache\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:41.933911 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:41.933885 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5990b367-6bd6-483b-ab76-35375b015cbd-tls-certs\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:41.933981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:41.933966 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-tmp-dir\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:41.934051 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:41.934035 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9d8k\" (UniqueName: \"kubernetes.io/projected/5990b367-6bd6-483b-ab76-35375b015cbd-kube-api-access-h9d8k\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:42.038289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:42.035882 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-tmp-dir\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:42.038289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:42.035964 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9d8k\" (UniqueName: \"kubernetes.io/projected/5990b367-6bd6-483b-ab76-35375b015cbd-kube-api-access-h9d8k\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:42.038289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:42.036016 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-home\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:42.038289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:42.036039 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-dshm\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:42.038289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:42.036126 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-model-cache\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:42.038289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:42.036207 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5990b367-6bd6-483b-ab76-35375b015cbd-tls-certs\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:42.062238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:42.058962 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-home\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:42.062238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:42.059117 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-model-cache\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:42.062238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:42.059268 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-tmp-dir\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:42.082019 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:42.079941 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-dshm\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:42.082019 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:42.081915 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9d8k\" (UniqueName: \"kubernetes.io/projected/5990b367-6bd6-483b-ab76-35375b015cbd-kube-api-access-h9d8k\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:42.085652 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:42.084585 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5990b367-6bd6-483b-ab76-35375b015cbd-tls-certs\") pod \"distinct-w2-aaf467-kserve-987d7768d-hjkcb\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:42.113632 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:42.113603 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:42.374691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:42.374660 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:42.421544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:42.421496 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb"] Oct 30 06:09:42.424363 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:09:42.424329 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5990b367_6bd6_483b_ab76_35375b015cbd.slice/crio-3a075484a4a33e3bb108552bb37dc425f212be7d917c6597d3e6dc690be5f9f5 WatchSource:0}: Error finding container 3a075484a4a33e3bb108552bb37dc425f212be7d917c6597d3e6dc690be5f9f5: Status 404 returned error can't find the container with id 3a075484a4a33e3bb108552bb37dc425f212be7d917c6597d3e6dc690be5f9f5 Oct 30 06:09:43.204981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:43.204945 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" event={"ID":"5990b367-6bd6-483b-ab76-35375b015cbd","Type":"ContainerStarted","Data":"fe0eff16cc7e522dc908c92eecc7e21c73811dc06956cbddc6906e8009907acf"} Oct 30 06:09:43.205404 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:43.204987 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" event={"ID":"5990b367-6bd6-483b-ab76-35375b015cbd","Type":"ContainerStarted","Data":"3a075484a4a33e3bb108552bb37dc425f212be7d917c6597d3e6dc690be5f9f5"} Oct 30 06:09:43.248062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:43.248012 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" podStartSLOduration=2.247999917 podStartE2EDuration="2.247999917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:43.245807471 +0000 UTC m=+949.754827371" watchObservedRunningTime="2026-10-30 06:09:43.247999917 +0000 UTC m=+949.757019803" Oct 30 06:09:44.114385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:44.114359 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:44.220167 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:44.220141 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:44.366757 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:44.366681 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:46.001583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:46.001551 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:46.387987 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:46.387958 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:46.400256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:46.400207 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:47.236537 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:47.236509 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 06:09:47.836842 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:47.836813 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:49.628089 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:49.628045 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:51.344487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:51.344453 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:52.114384 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:52.114331 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:52.128801 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:52.128768 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:52.277775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:52.277749 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:09:53.233982 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:53.233949 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:55.095162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:55.095088 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:55.393026 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:55.392998 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:56.938068 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:56.938034 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:58.085234 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:58.085182 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:58.747475 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:58.747442 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:09:59.073787 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:09:59.073699 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:00.222647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:00.222612 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:00.734833 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:00.734798 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:02.929296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:02.929265 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:03.961379 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:03.961349 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn"] Oct 30 06:10:03.966569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:03.966536 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:03.995619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:03.995588 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-eaee88\"/\"distinct-w4-eaee88-kserve-self-signed-certs\"" Oct 30 06:10:03.996411 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:03.996383 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn"] Oct 30 06:10:04.062697 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.062665 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:04.071043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.071018 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-tmp-dir\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.071185 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.071063 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-model-cache\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.071185 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.071106 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w94zl\" (UniqueName: \"kubernetes.io/projected/8e38cac7-3426-497c-8b84-e7552169bb7c-kube-api-access-w94zl\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.071305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.071281 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-dshm\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.071367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.071323 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e38cac7-3426-497c-8b84-e7552169bb7c-tls-certs\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.071426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.071408 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-home\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.142791 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:10:04.142752 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:10:04.174939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.172700 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-tmp-dir\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.174939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.172775 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-model-cache\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.174939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.172855 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w94zl\" (UniqueName: \"kubernetes.io/projected/8e38cac7-3426-497c-8b84-e7552169bb7c-kube-api-access-w94zl\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.174939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.172891 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-dshm\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.174939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.172938 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e38cac7-3426-497c-8b84-e7552169bb7c-tls-certs\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.174939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.172975 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-home\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.174939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.173478 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-home\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.174939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.173928 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-tmp-dir\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.174939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.174195 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-model-cache\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.182023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.181967 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-dshm\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.202152 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.202111 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e38cac7-3426-497c-8b84-e7552169bb7c-tls-certs\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.202383 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.202327 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w94zl\" (UniqueName: \"kubernetes.io/projected/8e38cac7-3426-497c-8b84-e7552169bb7c-kube-api-access-w94zl\") pod \"distinct-w4-eaee88-kserve-7b697c9b57-f58hn\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.281925 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.281828 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:04.591547 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.591456 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn"] Oct 30 06:10:04.595274 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:10:04.595248 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e38cac7_3426_497c_8b84_e7552169bb7c.slice/crio-6ca143acc2df8e646a79e47bd3637351d1d433956641f65a671122ce3eba885b WatchSource:0}: Error finding container 6ca143acc2df8e646a79e47bd3637351d1d433956641f65a671122ce3eba885b: Status 404 returned error can't find the container with id 6ca143acc2df8e646a79e47bd3637351d1d433956641f65a671122ce3eba885b Oct 30 06:10:04.738665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:04.738630 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:05.338432 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:05.338395 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" event={"ID":"8e38cac7-3426-497c-8b84-e7552169bb7c","Type":"ContainerStarted","Data":"bdb7708463cbed7a80c3e23d25870c524d1633ab0ae4c046887bb8e6aae46fed"} Oct 30 06:10:05.338843 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:05.338440 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" event={"ID":"8e38cac7-3426-497c-8b84-e7552169bb7c","Type":"ContainerStarted","Data":"6ca143acc2df8e646a79e47bd3637351d1d433956641f65a671122ce3eba885b"} Oct 30 06:10:05.383788 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:05.383734 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" podStartSLOduration=2.383720731 podStartE2EDuration="2.383720731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:05.381361112 +0000 UTC m=+971.890381000" watchObservedRunningTime="2026-10-30 06:10:05.383720731 +0000 UTC m=+971.892740660" Oct 30 06:10:06.282696 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:06.282666 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:06.537507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:06.537418 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:07.062968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:07.062934 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:08.353617 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:08.353581 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:10.149170 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:10.149133 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:11.931346 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:11.931313 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:13.960566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:13.960526 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:14.284683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:14.284594 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:14.318868 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:14.318833 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:14.417517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:14.417492 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:10:15.854622 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:15.854583 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:17.303604 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:17.303569 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:17.814884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:17.814853 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:19.637082 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:19.637052 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:21.431545 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:21.431507 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:23.339270 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:23.339236 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:25.154554 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:25.154516 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:25.300841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:25.300777 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:26.939917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:26.939877 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:28.749021 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:28.748987 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:30.432232 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:30.432184 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:31.558337 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:31.558306 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:32.277511 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:32.277479 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:34.135959 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:34.135888 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:34.141397 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:10:34.141195 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:10:35.923834 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:35.923807 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:37.739404 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:37.739371 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:39.545695 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:39.545654 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:41.315863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:41.315820 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:41.462550 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:41.462521 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:43.057002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:43.056907 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:45.063236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:45.059662 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:45.337764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:45.337673 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:46.923563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:46.923530 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:48.875581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:48.875545 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:49.807312 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:49.807278 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx"] Oct 30 06:10:49.811280 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:49.811261 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:49.841415 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:49.841389 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx"] Oct 30 06:10:49.904963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:49.904935 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa497a5-ee9b-4672-b80b-914b3ac99692-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:49.905360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:49.904971 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:49.905360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:49.904995 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:49.905360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:49.905140 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:49.905360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:49.905182 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:49.905360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:49.905224 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:49.905360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:49.905255 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:49.905360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:49.905347 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:49.905598 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:49.905382 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8pj8\" (UniqueName: \"kubernetes.io/projected/eaa497a5-ee9b-4672-b80b-914b3ac99692-kube-api-access-d8pj8\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.006751 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.006712 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.006900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.006765 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.006900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.006881 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.007085 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.006913 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.007085 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.007018 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.007085 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.007075 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8pj8\" (UniqueName: \"kubernetes.io/projected/eaa497a5-ee9b-4672-b80b-914b3ac99692-kube-api-access-d8pj8\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.007302 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.007112 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa497a5-ee9b-4672-b80b-914b3ac99692-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.007302 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.007145 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.007302 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.007175 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.007461 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.007319 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.007616 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.007596 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.007725 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.007702 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.007910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.007893 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa497a5-ee9b-4672-b80b-914b3ac99692-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.007951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.007904 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.009177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.009153 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.009397 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.009379 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.030480 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.030451 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.030603 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.030569 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8pj8\" (UniqueName: \"kubernetes.io/projected/eaa497a5-ee9b-4672-b80b-914b3ac99692-kube-api-access-d8pj8\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.126384 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.126359 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:50.445548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.445523 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx"] Oct 30 06:10:50.447522 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:10:50.447496 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa497a5_ee9b_4672_b80b_914b3ac99692.slice/crio-45e0ab622a54c29ed2b25416d1a7e4016fc233577889ab6c5914dbb0d3971751 WatchSource:0}: Error finding container 45e0ab622a54c29ed2b25416d1a7e4016fc233577889ab6c5914dbb0d3971751: Status 404 returned error can't find the container with id 45e0ab622a54c29ed2b25416d1a7e4016fc233577889ab6c5914dbb0d3971751 Oct 30 06:10:50.450098 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.450065 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:50.450184 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.450140 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:50.450184 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.450168 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:50.640382 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.640349 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" event={"ID":"eaa497a5-ee9b-4672-b80b-914b3ac99692","Type":"ContainerStarted","Data":"31a4023a9e5b3832b2bf5999a37f969a867e977e9f159df6c15d518a7d602ab0"} Oct 30 06:10:50.640565 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.640402 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" event={"ID":"eaa497a5-ee9b-4672-b80b-914b3ac99692","Type":"ContainerStarted","Data":"45e0ab622a54c29ed2b25416d1a7e4016fc233577889ab6c5914dbb0d3971751"} Oct 30 06:10:50.685512 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.685462 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" podStartSLOduration=1.6854436750000001 podStartE2EDuration="1.685443675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:50.68442003 +0000 UTC m=+1017.193439918" watchObservedRunningTime="2026-10-30 06:10:50.685443675 +0000 UTC m=+1017.194463561" Oct 30 06:10:50.825709 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:50.825613 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:51.126667 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.126633 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:51.127840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.127817 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 06:10:51.127905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.127882 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" podUID="eaa497a5-ee9b-4672-b80b-914b3ac99692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 06:10:51.218650 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.218619 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg"] Oct 30 06:10:51.222830 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.222814 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" Oct 30 06:10:51.265244 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.255700 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg"] Oct 30 06:10:51.265244 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.255862 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:10:51.265244 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.255997 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:10:51.329173 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.329140 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpbg7\" (UniqueName: \"kubernetes.io/projected/8168c11e-ee79-4ffb-a53f-4f10563ff965-kube-api-access-mpbg7\") pod \"maas-api-revoke-sub-7ca66358-jwlrg\" (UID: \"8168c11e-ee79-4ffb-a53f-4f10563ff965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" Oct 30 06:10:51.329334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.329286 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8168c11e-ee79-4ffb-a53f-4f10563ff965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ca66358-jwlrg\" (UID: \"8168c11e-ee79-4ffb-a53f-4f10563ff965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" Oct 30 06:10:51.329388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.329334 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8168c11e-ee79-4ffb-a53f-4f10563ff965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ca66358-jwlrg\" (UID: \"8168c11e-ee79-4ffb-a53f-4f10563ff965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" Oct 30 06:10:51.430664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.430575 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbg7\" (UniqueName: \"kubernetes.io/projected/8168c11e-ee79-4ffb-a53f-4f10563ff965-kube-api-access-mpbg7\") pod \"maas-api-revoke-sub-7ca66358-jwlrg\" (UID: \"8168c11e-ee79-4ffb-a53f-4f10563ff965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" Oct 30 06:10:51.430664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.430657 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8168c11e-ee79-4ffb-a53f-4f10563ff965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ca66358-jwlrg\" (UID: \"8168c11e-ee79-4ffb-a53f-4f10563ff965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" Oct 30 06:10:51.430849 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.430699 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8168c11e-ee79-4ffb-a53f-4f10563ff965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ca66358-jwlrg\" (UID: \"8168c11e-ee79-4ffb-a53f-4f10563ff965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" Oct 30 06:10:51.431373 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.431355 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8168c11e-ee79-4ffb-a53f-4f10563ff965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ca66358-jwlrg\" (UID: \"8168c11e-ee79-4ffb-a53f-4f10563ff965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" Oct 30 06:10:51.467099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.467051 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8168c11e-ee79-4ffb-a53f-4f10563ff965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ca66358-jwlrg\" (UID: \"8168c11e-ee79-4ffb-a53f-4f10563ff965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" Oct 30 06:10:51.470442 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.470419 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpbg7\" (UniqueName: \"kubernetes.io/projected/8168c11e-ee79-4ffb-a53f-4f10563ff965-kube-api-access-mpbg7\") pod \"maas-api-revoke-sub-7ca66358-jwlrg\" (UID: \"8168c11e-ee79-4ffb-a53f-4f10563ff965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" Oct 30 06:10:51.562591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.562555 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" Oct 30 06:10:51.865841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:51.865816 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg"] Oct 30 06:10:51.868360 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:10:51.868333 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8168c11e_ee79_4ffb_a53f_4f10563ff965.slice/crio-65672245fe195bdeadc8f2d06fdea51ccfa6b964efdbbd66aad4603e4cbbe6b8 WatchSource:0}: Error finding container 65672245fe195bdeadc8f2d06fdea51ccfa6b964efdbbd66aad4603e4cbbe6b8: Status 404 returned error can't find the container with id 65672245fe195bdeadc8f2d06fdea51ccfa6b964efdbbd66aad4603e4cbbe6b8 Oct 30 06:10:52.126900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:52.126829 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 06:10:52.126900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:52.126882 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" podUID="eaa497a5-ee9b-4672-b80b-914b3ac99692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 06:10:52.672018 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:52.671986 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:52.676842 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:52.676811 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" event={"ID":"8168c11e-ee79-4ffb-a53f-4f10563ff965","Type":"ContainerStarted","Data":"4879cd429720feca39972993aac143864b94d76f02432c70c2b15d8d2e91dfc8"} Oct 30 06:10:52.676972 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:52.676868 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" event={"ID":"8168c11e-ee79-4ffb-a53f-4f10563ff965","Type":"ContainerStarted","Data":"65672245fe195bdeadc8f2d06fdea51ccfa6b964efdbbd66aad4603e4cbbe6b8"} Oct 30 06:10:52.718067 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:52.718015 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" podStartSLOduration=1.717996398 podStartE2EDuration="1.717996398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:52.716177637 +0000 UTC m=+1019.225197536" watchObservedRunningTime="2026-10-30 06:10:52.717996398 +0000 UTC m=+1019.227016287" Oct 30 06:10:53.127718 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:53.127686 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 06:10:53.128080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:53.127737 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" podUID="eaa497a5-ee9b-4672-b80b-914b3ac99692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 06:10:53.683550 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:53.683518 6999 generic.go:356] "Generic (PLEG): container finished" podID="8168c11e-ee79-4ffb-a53f-4f10563ff965" containerID="4879cd429720feca39972993aac143864b94d76f02432c70c2b15d8d2e91dfc8" exitCode=0 Oct 30 06:10:53.683717 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:53.683604 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" event={"ID":"8168c11e-ee79-4ffb-a53f-4f10563ff965","Type":"ContainerDied","Data":"4879cd429720feca39972993aac143864b94d76f02432c70c2b15d8d2e91dfc8"} Oct 30 06:10:54.127117 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:54.127084 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 06:10:54.127321 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:54.127133 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" podUID="eaa497a5-ee9b-4672-b80b-914b3ac99692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 06:10:54.660816 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:54.660781 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:54.889808 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:54.889752 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" Oct 30 06:10:54.998275 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:54.998246 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8168c11e-ee79-4ffb-a53f-4f10563ff965-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8168c11e-ee79-4ffb-a53f-4f10563ff965-maas-api-openshift-ca\") pod \"8168c11e-ee79-4ffb-a53f-4f10563ff965\" (UID: \"8168c11e-ee79-4ffb-a53f-4f10563ff965\") " Oct 30 06:10:54.998479 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:54.998286 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8168c11e-ee79-4ffb-a53f-4f10563ff965-kube-api-access-mpbg7\" (UniqueName: \"kubernetes.io/projected/8168c11e-ee79-4ffb-a53f-4f10563ff965-kube-api-access-mpbg7\") pod \"8168c11e-ee79-4ffb-a53f-4f10563ff965\" (UID: \"8168c11e-ee79-4ffb-a53f-4f10563ff965\") " Oct 30 06:10:54.998479 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:54.998368 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8168c11e-ee79-4ffb-a53f-4f10563ff965-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8168c11e-ee79-4ffb-a53f-4f10563ff965-maas-api-xks-ca\") pod \"8168c11e-ee79-4ffb-a53f-4f10563ff965\" (UID: \"8168c11e-ee79-4ffb-a53f-4f10563ff965\") " Oct 30 06:10:54.998699 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:54.998669 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8168c11e-ee79-4ffb-a53f-4f10563ff965-maas-api-openshift-ca" pod "8168c11e-ee79-4ffb-a53f-4f10563ff965" (UID: "8168c11e-ee79-4ffb-a53f-4f10563ff965"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:55.000638 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:55.000609 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8168c11e-ee79-4ffb-a53f-4f10563ff965-kube-api-access-mpbg7" pod "8168c11e-ee79-4ffb-a53f-4f10563ff965" (UID: "8168c11e-ee79-4ffb-a53f-4f10563ff965"). InnerVolumeSpecName "kube-api-access-mpbg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:55.000638 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:55.000629 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8168c11e-ee79-4ffb-a53f-4f10563ff965-maas-api-xks-ca" pod "8168c11e-ee79-4ffb-a53f-4f10563ff965" (UID: "8168c11e-ee79-4ffb-a53f-4f10563ff965"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:10:55.099970 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:55.099934 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8168c11e-ee79-4ffb-a53f-4f10563ff965-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:10:55.099970 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:55.099964 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8168c11e-ee79-4ffb-a53f-4f10563ff965-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:10:55.099970 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:55.099974 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpbg7\" (UniqueName: \"kubernetes.io/projected/8168c11e-ee79-4ffb-a53f-4f10563ff965-kube-api-access-mpbg7\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:10:55.127525 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:55.127496 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 06:10:55.127636 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:55.127547 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" podUID="eaa497a5-ee9b-4672-b80b-914b3ac99692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 06:10:55.695351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:55.695325 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" Oct 30 06:10:55.695737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:55.695345 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg" event={"ID":"8168c11e-ee79-4ffb-a53f-4f10563ff965","Type":"ContainerDied","Data":"65672245fe195bdeadc8f2d06fdea51ccfa6b964efdbbd66aad4603e4cbbe6b8"} Oct 30 06:10:55.695737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:55.695391 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65672245fe195bdeadc8f2d06fdea51ccfa6b964efdbbd66aad4603e4cbbe6b8" Oct 30 06:10:56.127429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:56.127397 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 06:10:56.127617 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:56.127459 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" podUID="eaa497a5-ee9b-4672-b80b-914b3ac99692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 06:10:56.548049 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:56.547941 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:57.127486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:57.127444 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 06:10:57.127983 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:57.127496 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" podUID="eaa497a5-ee9b-4672-b80b-914b3ac99692" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 06:10:58.495848 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:58.495811 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:58.766926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:58.766854 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:10:59.106817 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:59.106734 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:59.107085 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:59.107059 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:59.108022 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:59.108000 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 06:10:59.221030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:10:59.220994 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:00.307560 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:00.307533 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:00.629289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:00.629257 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:01.780693 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:01.780660 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:02.537679 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:02.537652 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:03.739600 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.739566 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j"] Oct 30 06:11:03.740063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.740050 6999 cpu_manager.go:414] "containerMap: removing container" podUID="8168c11e-ee79-4ffb-a53f-4f10563ff965" containerName="revoke-keys" Oct 30 06:11:03.740108 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.740064 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8168c11e-ee79-4ffb-a53f-4f10563ff965" containerName="revoke-keys" Oct 30 06:11:03.740195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.740173 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="8168c11e-ee79-4ffb-a53f-4f10563ff965" containerName="revoke-keys" Oct 30 06:11:03.743444 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.743429 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" Oct 30 06:11:03.763322 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.763291 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j"] Oct 30 06:11:03.772559 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.772526 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:03.772729 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.772569 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:11:03.804122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.804088 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22lt9\" (UniqueName: \"kubernetes.io/projected/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-kube-api-access-22lt9\") pod \"maas-api-revoke-sub-482cb1aa-mxt4j\" (UID: \"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" Oct 30 06:11:03.804318 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.804132 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-482cb1aa-mxt4j\" (UID: \"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" Oct 30 06:11:03.804318 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.804160 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-482cb1aa-mxt4j\" (UID: \"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" Oct 30 06:11:03.905577 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.905543 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22lt9\" (UniqueName: \"kubernetes.io/projected/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-kube-api-access-22lt9\") pod \"maas-api-revoke-sub-482cb1aa-mxt4j\" (UID: \"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" Oct 30 06:11:03.905577 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.905584 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-482cb1aa-mxt4j\" (UID: \"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" Oct 30 06:11:03.905843 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.905602 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-482cb1aa-mxt4j\" (UID: \"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" Oct 30 06:11:03.906243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.906189 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-482cb1aa-mxt4j\" (UID: \"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" Oct 30 06:11:03.908075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.908055 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-maas-api-xks-ca\") pod \"maas-api-revoke-sub-482cb1aa-mxt4j\" (UID: \"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" Oct 30 06:11:03.929478 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:03.929449 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22lt9\" (UniqueName: \"kubernetes.io/projected/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-kube-api-access-22lt9\") pod \"maas-api-revoke-sub-482cb1aa-mxt4j\" (UID: \"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" Oct 30 06:11:04.056774 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.056690 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" Oct 30 06:11:04.164081 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:11:04.164042 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:11:04.365685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.365662 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j"] Oct 30 06:11:04.367735 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:11:04.367706 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e4084a_aabb_49f7_bfbb_6a0d4f14e202.slice/crio-eabe15e92df822ce45509dde47c93501802ffc653d3f87e8e84533f00d29db64 WatchSource:0}: Error finding container eabe15e92df822ce45509dde47c93501802ffc653d3f87e8e84533f00d29db64: Status 404 returned error can't find the container with id eabe15e92df822ce45509dde47c93501802ffc653d3f87e8e84533f00d29db64 Oct 30 06:11:04.417324 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.417296 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc"] Oct 30 06:11:04.422623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.422604 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.449231 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.449183 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc"] Oct 30 06:11:04.452164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.452108 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:04.512702 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.512668 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fa2534f-7589-4a20-b321-10df418b86c5-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.512850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.512732 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fa2534f-7589-4a20-b321-10df418b86c5-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.512850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.512814 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.512850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.512846 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.512984 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.512878 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnjj2\" (UniqueName: \"kubernetes.io/projected/0fa2534f-7589-4a20-b321-10df418b86c5-kube-api-access-bnjj2\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.512984 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.512900 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.513050 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.513019 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.513088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.513055 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.513178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.513158 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fa2534f-7589-4a20-b321-10df418b86c5-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.614347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.614316 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fa2534f-7589-4a20-b321-10df418b86c5-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.614499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.614356 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fa2534f-7589-4a20-b321-10df418b86c5-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.614499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.614418 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fa2534f-7589-4a20-b321-10df418b86c5-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.614499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.614469 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.614694 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.614503 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.614694 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.614544 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnjj2\" (UniqueName: \"kubernetes.io/projected/0fa2534f-7589-4a20-b321-10df418b86c5-kube-api-access-bnjj2\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.614694 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.614575 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.614694 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.614646 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.614694 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.614685 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.614993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.614970 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.615062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.615017 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.615146 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.615115 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.615146 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.615138 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fa2534f-7589-4a20-b321-10df418b86c5-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.615146 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.615169 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.616867 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.616844 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.617007 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.616992 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fa2534f-7589-4a20-b321-10df418b86c5-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.639814 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.639792 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fa2534f-7589-4a20-b321-10df418b86c5-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.640102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.640085 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnjj2\" (UniqueName: \"kubernetes.io/projected/0fa2534f-7589-4a20-b321-10df418b86c5-kube-api-access-bnjj2\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.736581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.736554 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:04.757071 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.757045 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" event={"ID":"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202","Type":"ContainerStarted","Data":"e3226bfcde8afc25ee2157c5bf6cbb6ff002c226741c389e1d266308f8b7d211"} Oct 30 06:11:04.757495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.757095 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" event={"ID":"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202","Type":"ContainerStarted","Data":"eabe15e92df822ce45509dde47c93501802ffc653d3f87e8e84533f00d29db64"} Oct 30 06:11:04.799090 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:04.798726 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" podStartSLOduration=1.798706856 podStartE2EDuration="1.798706856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:04.797489698 +0000 UTC m=+1031.306509588" watchObservedRunningTime="2026-10-30 06:11:04.798706856 +0000 UTC m=+1031.307726748" Oct 30 06:11:05.066282 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:05.066250 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc"] Oct 30 06:11:05.067363 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:11:05.067325 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa2534f_7589_4a20_b321_10df418b86c5.slice/crio-ab62d839f66ee379cb0952002e4c5e56c343cd3b13c05afd49b0a9396a13e607 WatchSource:0}: Error finding container ab62d839f66ee379cb0952002e4c5e56c343cd3b13c05afd49b0a9396a13e607: Status 404 returned error can't find the container with id ab62d839f66ee379cb0952002e4c5e56c343cd3b13c05afd49b0a9396a13e607 Oct 30 06:11:05.069874 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:05.069827 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:05.070036 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:05.069993 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:05.070146 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:05.070123 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:05.766812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:05.766779 6999 generic.go:356] "Generic (PLEG): container finished" podID="c2e4084a-aabb-49f7-bfbb-6a0d4f14e202" containerID="e3226bfcde8afc25ee2157c5bf6cbb6ff002c226741c389e1d266308f8b7d211" exitCode=0 Oct 30 06:11:05.767283 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:05.766859 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" event={"ID":"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202","Type":"ContainerDied","Data":"e3226bfcde8afc25ee2157c5bf6cbb6ff002c226741c389e1d266308f8b7d211"} Oct 30 06:11:05.768616 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:05.768592 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" event={"ID":"0fa2534f-7589-4a20-b321-10df418b86c5","Type":"ContainerStarted","Data":"b612bbc37ba6b8f11cff71b7f827f7c230a12b9ef80c99d71a1c188247de47b4"} Oct 30 06:11:05.768724 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:05.768636 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" event={"ID":"0fa2534f-7589-4a20-b321-10df418b86c5","Type":"ContainerStarted","Data":"ab62d839f66ee379cb0952002e4c5e56c343cd3b13c05afd49b0a9396a13e607"} Oct 30 06:11:05.860201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:05.860129 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" podStartSLOduration=1.860106986 podStartE2EDuration="1.860106986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:05.85788268 +0000 UTC m=+1032.366902568" watchObservedRunningTime="2026-10-30 06:11:05.860106986 +0000 UTC m=+1032.369126875" Oct 30 06:11:06.383951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:06.383913 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:06.739295 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:06.738463 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:06.758246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:06.757305 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 06:11:06.758246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:06.757361 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" podUID="0fa2534f-7589-4a20-b321-10df418b86c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 06:11:06.933658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:06.933630 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" Oct 30 06:11:07.060294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:07.059602 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-kube-api-access-22lt9\" (UniqueName: \"kubernetes.io/projected/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-kube-api-access-22lt9\") pod \"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202\" (UID: \"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202\") " Oct 30 06:11:07.060294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:07.059653 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-maas-api-openshift-ca\") pod \"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202\" (UID: \"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202\") " Oct 30 06:11:07.060294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:07.059909 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-maas-api-xks-ca\") pod \"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202\" (UID: \"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202\") " Oct 30 06:11:07.060609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:07.060570 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-maas-api-openshift-ca" pod "c2e4084a-aabb-49f7-bfbb-6a0d4f14e202" (UID: "c2e4084a-aabb-49f7-bfbb-6a0d4f14e202"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:07.065392 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:07.065359 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-maas-api-xks-ca" pod "c2e4084a-aabb-49f7-bfbb-6a0d4f14e202" (UID: "c2e4084a-aabb-49f7-bfbb-6a0d4f14e202"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:07.065845 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:07.065819 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-kube-api-access-22lt9" pod "c2e4084a-aabb-49f7-bfbb-6a0d4f14e202" (UID: "c2e4084a-aabb-49f7-bfbb-6a0d4f14e202"). InnerVolumeSpecName "kube-api-access-22lt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:07.161587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:07.161519 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:11:07.161587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:07.161544 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22lt9\" (UniqueName: \"kubernetes.io/projected/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-kube-api-access-22lt9\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:11:07.161587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:07.161555 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:11:07.739722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:07.739305 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 06:11:07.739722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:07.739360 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" podUID="0fa2534f-7589-4a20-b321-10df418b86c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 06:11:07.780887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:07.780863 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" Oct 30 06:11:07.781071 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:07.780944 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j" event={"ID":"c2e4084a-aabb-49f7-bfbb-6a0d4f14e202","Type":"ContainerDied","Data":"eabe15e92df822ce45509dde47c93501802ffc653d3f87e8e84533f00d29db64"} Oct 30 06:11:07.781071 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:07.780997 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eabe15e92df822ce45509dde47c93501802ffc653d3f87e8e84533f00d29db64" Oct 30 06:11:08.210937 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:08.210905 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:08.746238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:08.738527 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 06:11:08.746238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:08.738587 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" podUID="0fa2534f-7589-4a20-b321-10df418b86c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 06:11:09.750239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:09.739523 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 06:11:09.750239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:09.739584 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" podUID="0fa2534f-7589-4a20-b321-10df418b86c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 06:11:10.164839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:10.164805 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:10.752241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:10.749544 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 06:11:10.752241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:10.749602 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" podUID="0fa2534f-7589-4a20-b321-10df418b86c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 06:11:11.736855 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:11.736821 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 06:11:11.737021 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:11.736880 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" podUID="0fa2534f-7589-4a20-b321-10df418b86c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 06:11:12.049946 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:12.049864 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:12.696087 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:12.696055 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:12.743513 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:12.743480 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:12.744347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:12.744319 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:12.745312 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:12.745293 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:11:14.244447 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:14.243870 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" podUID="987180e9-4669-4740-b2df-effebab90159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:14.278442 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:14.277435 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" podUID="5990b367-6bd6-483b-ab76-35375b015cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:14.413349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:14.413306 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" podUID="8e38cac7-3426-497c-8b84-e7552169bb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:14.475420 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:14.475366 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" podUID="79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:14.697236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:14.697080 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" podUID="b2701c51-6377-411c-9924-715ff0765564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:15.236307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:15.236271 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:15.434127 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:15.434086 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:15.862017 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:15.861990 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn"] Oct 30 06:11:15.862534 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:15.862518 6999 cpu_manager.go:414] "containerMap: removing container" podUID="c2e4084a-aabb-49f7-bfbb-6a0d4f14e202" containerName="revoke-keys" Oct 30 06:11:15.862534 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:15.862535 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e4084a-aabb-49f7-bfbb-6a0d4f14e202" containerName="revoke-keys" Oct 30 06:11:15.862678 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:15.862655 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2e4084a-aabb-49f7-bfbb-6a0d4f14e202" containerName="revoke-keys" Oct 30 06:11:15.866236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:15.866202 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" Oct 30 06:11:15.889236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:15.889181 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn"] Oct 30 06:11:15.894996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:15.894972 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:15.894996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:15.894987 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:11:15.989863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:15.989829 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5x78\" (UniqueName: \"kubernetes.io/projected/26908887-9f3f-4e52-9c93-799e9581a075-kube-api-access-s5x78\") pod \"maas-api-revoke-sub-e150dea1-dktxn\" (UID: \"26908887-9f3f-4e52-9c93-799e9581a075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" Oct 30 06:11:15.990029 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:15.989871 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26908887-9f3f-4e52-9c93-799e9581a075-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e150dea1-dktxn\" (UID: \"26908887-9f3f-4e52-9c93-799e9581a075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" Oct 30 06:11:15.990029 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:15.989985 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26908887-9f3f-4e52-9c93-799e9581a075-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e150dea1-dktxn\" (UID: \"26908887-9f3f-4e52-9c93-799e9581a075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" Oct 30 06:11:16.090912 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:16.090882 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5x78\" (UniqueName: \"kubernetes.io/projected/26908887-9f3f-4e52-9c93-799e9581a075-kube-api-access-s5x78\") pod \"maas-api-revoke-sub-e150dea1-dktxn\" (UID: \"26908887-9f3f-4e52-9c93-799e9581a075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" Oct 30 06:11:16.091083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:16.090922 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26908887-9f3f-4e52-9c93-799e9581a075-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e150dea1-dktxn\" (UID: \"26908887-9f3f-4e52-9c93-799e9581a075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" Oct 30 06:11:16.091083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:16.090960 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26908887-9f3f-4e52-9c93-799e9581a075-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e150dea1-dktxn\" (UID: \"26908887-9f3f-4e52-9c93-799e9581a075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" Oct 30 06:11:16.091887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:16.091865 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26908887-9f3f-4e52-9c93-799e9581a075-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e150dea1-dktxn\" (UID: \"26908887-9f3f-4e52-9c93-799e9581a075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" Oct 30 06:11:16.093387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:16.093369 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26908887-9f3f-4e52-9c93-799e9581a075-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e150dea1-dktxn\" (UID: \"26908887-9f3f-4e52-9c93-799e9581a075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" Oct 30 06:11:16.114156 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:16.114098 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5x78\" (UniqueName: \"kubernetes.io/projected/26908887-9f3f-4e52-9c93-799e9581a075-kube-api-access-s5x78\") pod \"maas-api-revoke-sub-e150dea1-dktxn\" (UID: \"26908887-9f3f-4e52-9c93-799e9581a075\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" Oct 30 06:11:16.183258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:16.183229 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" Oct 30 06:11:16.517356 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:16.517330 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn"] Oct 30 06:11:16.518519 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:11:16.518493 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26908887_9f3f_4e52_9c93_799e9581a075.slice/crio-ae7e3f870b20e1acc51c5b2c6fe51f543a4a8538368ea32a028856f90e4217c2 WatchSource:0}: Error finding container ae7e3f870b20e1acc51c5b2c6fe51f543a4a8538368ea32a028856f90e4217c2: Status 404 returned error can't find the container with id ae7e3f870b20e1acc51c5b2c6fe51f543a4a8538368ea32a028856f90e4217c2 Oct 30 06:11:16.855750 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:16.855718 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" event={"ID":"26908887-9f3f-4e52-9c93-799e9581a075","Type":"ContainerStarted","Data":"ae7e3f870b20e1acc51c5b2c6fe51f543a4a8538368ea32a028856f90e4217c2"} Oct 30 06:11:17.361154 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:17.361118 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:17.863752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:17.863715 6999 generic.go:356] "Generic (PLEG): container finished" podID="26908887-9f3f-4e52-9c93-799e9581a075" containerID="ae54d1ab9e7efd07da317134084c05b6dd77ed95d56561b05771f0a5d11cf4a5" exitCode=0 Oct 30 06:11:17.864192 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:17.863794 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" event={"ID":"26908887-9f3f-4e52-9c93-799e9581a075","Type":"ContainerDied","Data":"ae54d1ab9e7efd07da317134084c05b6dd77ed95d56561b05771f0a5d11cf4a5"} Oct 30 06:11:19.010974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:19.010954 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" Oct 30 06:11:19.124011 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:19.123949 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26908887-9f3f-4e52-9c93-799e9581a075-kube-api-access-s5x78\" (UniqueName: \"kubernetes.io/projected/26908887-9f3f-4e52-9c93-799e9581a075-kube-api-access-s5x78\") pod \"26908887-9f3f-4e52-9c93-799e9581a075\" (UID: \"26908887-9f3f-4e52-9c93-799e9581a075\") " Oct 30 06:11:19.124167 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:19.124029 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26908887-9f3f-4e52-9c93-799e9581a075-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26908887-9f3f-4e52-9c93-799e9581a075-maas-api-xks-ca\") pod \"26908887-9f3f-4e52-9c93-799e9581a075\" (UID: \"26908887-9f3f-4e52-9c93-799e9581a075\") " Oct 30 06:11:19.124258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:19.124177 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26908887-9f3f-4e52-9c93-799e9581a075-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26908887-9f3f-4e52-9c93-799e9581a075-maas-api-openshift-ca\") pod \"26908887-9f3f-4e52-9c93-799e9581a075\" (UID: \"26908887-9f3f-4e52-9c93-799e9581a075\") " Oct 30 06:11:19.124541 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:19.124515 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26908887-9f3f-4e52-9c93-799e9581a075-maas-api-openshift-ca" pod "26908887-9f3f-4e52-9c93-799e9581a075" (UID: "26908887-9f3f-4e52-9c93-799e9581a075"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:19.126122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:19.126101 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26908887-9f3f-4e52-9c93-799e9581a075-maas-api-xks-ca" pod "26908887-9f3f-4e52-9c93-799e9581a075" (UID: "26908887-9f3f-4e52-9c93-799e9581a075"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:19.126229 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:19.126136 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26908887-9f3f-4e52-9c93-799e9581a075-kube-api-access-s5x78" pod "26908887-9f3f-4e52-9c93-799e9581a075" (UID: "26908887-9f3f-4e52-9c93-799e9581a075"). InnerVolumeSpecName "kube-api-access-s5x78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:19.226204 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:19.226171 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26908887-9f3f-4e52-9c93-799e9581a075-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:11:19.226204 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:19.226202 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5x78\" (UniqueName: \"kubernetes.io/projected/26908887-9f3f-4e52-9c93-799e9581a075-kube-api-access-s5x78\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:11:19.226204 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:19.226230 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26908887-9f3f-4e52-9c93-799e9581a075-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:11:19.245347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:19.245313 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:19.876501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:19.876424 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" event={"ID":"26908887-9f3f-4e52-9c93-799e9581a075","Type":"ContainerDied","Data":"ae7e3f870b20e1acc51c5b2c6fe51f543a4a8538368ea32a028856f90e4217c2"} Oct 30 06:11:19.876501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:19.876449 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn" Oct 30 06:11:19.876501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:19.876470 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae7e3f870b20e1acc51c5b2c6fe51f543a4a8538368ea32a028856f90e4217c2" Oct 30 06:11:21.089102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:21.089061 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:22.926313 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:22.926282 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:24.492562 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:24.492531 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:24.636420 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:24.636391 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:24.840758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:24.840675 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:26.327236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.326328 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt"] Oct 30 06:11:26.327236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.327057 6999 cpu_manager.go:414] "containerMap: removing container" podUID="26908887-9f3f-4e52-9c93-799e9581a075" containerName="revoke-keys" Oct 30 06:11:26.327236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.327076 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26908887-9f3f-4e52-9c93-799e9581a075" containerName="revoke-keys" Oct 30 06:11:26.327837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.327299 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="26908887-9f3f-4e52-9c93-799e9581a075" containerName="revoke-keys" Oct 30 06:11:26.364239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.363589 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" Oct 30 06:11:26.364609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.363461 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt"] Oct 30 06:11:26.383783 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.383751 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:26.411323 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.409538 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:11:26.571344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.563855 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cbecf98-3905-4212-994c-447b3cae6827-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca9e7e78-zsqjt\" (UID: \"1cbecf98-3905-4212-994c-447b3cae6827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" Oct 30 06:11:26.571344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.563970 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cbecf98-3905-4212-994c-447b3cae6827-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca9e7e78-zsqjt\" (UID: \"1cbecf98-3905-4212-994c-447b3cae6827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" Oct 30 06:11:26.571344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.564035 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9brx9\" (UniqueName: \"kubernetes.io/projected/1cbecf98-3905-4212-994c-447b3cae6827-kube-api-access-9brx9\") pod \"maas-api-revoke-sub-ca9e7e78-zsqjt\" (UID: \"1cbecf98-3905-4212-994c-447b3cae6827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" Oct 30 06:11:26.672250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.668651 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cbecf98-3905-4212-994c-447b3cae6827-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca9e7e78-zsqjt\" (UID: \"1cbecf98-3905-4212-994c-447b3cae6827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" Oct 30 06:11:26.672250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.668743 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9brx9\" (UniqueName: \"kubernetes.io/projected/1cbecf98-3905-4212-994c-447b3cae6827-kube-api-access-9brx9\") pod \"maas-api-revoke-sub-ca9e7e78-zsqjt\" (UID: \"1cbecf98-3905-4212-994c-447b3cae6827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" Oct 30 06:11:26.672250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.668851 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cbecf98-3905-4212-994c-447b3cae6827-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca9e7e78-zsqjt\" (UID: \"1cbecf98-3905-4212-994c-447b3cae6827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" Oct 30 06:11:26.672250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.669727 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cbecf98-3905-4212-994c-447b3cae6827-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca9e7e78-zsqjt\" (UID: \"1cbecf98-3905-4212-994c-447b3cae6827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" Oct 30 06:11:26.707238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.701254 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cbecf98-3905-4212-994c-447b3cae6827-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca9e7e78-zsqjt\" (UID: \"1cbecf98-3905-4212-994c-447b3cae6827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" Oct 30 06:11:26.737887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.737859 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9brx9\" (UniqueName: \"kubernetes.io/projected/1cbecf98-3905-4212-994c-447b3cae6827-kube-api-access-9brx9\") pod \"maas-api-revoke-sub-ca9e7e78-zsqjt\" (UID: \"1cbecf98-3905-4212-994c-447b3cae6827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" Oct 30 06:11:26.801429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:26.801380 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" Oct 30 06:11:27.449247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:27.446313 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" podUID="8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:27.608248 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:11:27.607861 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cbecf98_3905_4212_994c_447b3cae6827.slice/crio-d1ca8bbcdeafecbe338a02441823776aa5d59d0516a005716778169430a2a815 WatchSource:0}: Error finding container d1ca8bbcdeafecbe338a02441823776aa5d59d0516a005716778169430a2a815: Status 404 returned error can't find the container with id d1ca8bbcdeafecbe338a02441823776aa5d59d0516a005716778169430a2a815 Oct 30 06:11:27.614169 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:27.614145 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt"] Oct 30 06:11:27.734426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:27.734378 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:11:27.734603 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:27.734443 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" podUID="a6aefe60-58f8-490e-9c3d-e062edf6e038" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:27.821062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:27.821033 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:27.960276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:27.960248 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" event={"ID":"1cbecf98-3905-4212-994c-447b3cae6827","Type":"ContainerStarted","Data":"d1ca8bbcdeafecbe338a02441823776aa5d59d0516a005716778169430a2a815"} Oct 30 06:11:28.967546 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:28.967462 6999 generic.go:356] "Generic (PLEG): container finished" podID="1cbecf98-3905-4212-994c-447b3cae6827" containerID="4b9cb6da74effafa84cd4ea321c6dcbfab221d75e634d48c9fd2b94d62be29e6" exitCode=0 Oct 30 06:11:28.967955 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:28.967540 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" event={"ID":"1cbecf98-3905-4212-994c-447b3cae6827","Type":"ContainerDied","Data":"4b9cb6da74effafa84cd4ea321c6dcbfab221d75e634d48c9fd2b94d62be29e6"} Oct 30 06:11:29.641994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:29.641947 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:30.124455 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:30.124431 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" Oct 30 06:11:30.148292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:30.148267 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cbecf98-3905-4212-994c-447b3cae6827-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cbecf98-3905-4212-994c-447b3cae6827-maas-api-xks-ca\") pod \"1cbecf98-3905-4212-994c-447b3cae6827\" (UID: \"1cbecf98-3905-4212-994c-447b3cae6827\") " Oct 30 06:11:30.148422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:30.148388 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cbecf98-3905-4212-994c-447b3cae6827-kube-api-access-9brx9\" (UniqueName: \"kubernetes.io/projected/1cbecf98-3905-4212-994c-447b3cae6827-kube-api-access-9brx9\") pod \"1cbecf98-3905-4212-994c-447b3cae6827\" (UID: \"1cbecf98-3905-4212-994c-447b3cae6827\") " Oct 30 06:11:30.148422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:30.148417 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cbecf98-3905-4212-994c-447b3cae6827-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cbecf98-3905-4212-994c-447b3cae6827-maas-api-openshift-ca\") pod \"1cbecf98-3905-4212-994c-447b3cae6827\" (UID: \"1cbecf98-3905-4212-994c-447b3cae6827\") " Oct 30 06:11:30.148798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:30.148778 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cbecf98-3905-4212-994c-447b3cae6827-maas-api-openshift-ca" pod "1cbecf98-3905-4212-994c-447b3cae6827" (UID: "1cbecf98-3905-4212-994c-447b3cae6827"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:30.150395 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:30.150372 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cbecf98-3905-4212-994c-447b3cae6827-maas-api-xks-ca" pod "1cbecf98-3905-4212-994c-447b3cae6827" (UID: "1cbecf98-3905-4212-994c-447b3cae6827"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:30.150587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:30.150574 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cbecf98-3905-4212-994c-447b3cae6827-kube-api-access-9brx9" pod "1cbecf98-3905-4212-994c-447b3cae6827" (UID: "1cbecf98-3905-4212-994c-447b3cae6827"). InnerVolumeSpecName "kube-api-access-9brx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:30.249999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:30.249920 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cbecf98-3905-4212-994c-447b3cae6827-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:11:30.249999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:30.249952 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9brx9\" (UniqueName: \"kubernetes.io/projected/1cbecf98-3905-4212-994c-447b3cae6827-kube-api-access-9brx9\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:11:30.249999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:30.249966 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cbecf98-3905-4212-994c-447b3cae6827-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:11:30.980486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:30.980454 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" Oct 30 06:11:30.980648 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:30.980480 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt" event={"ID":"1cbecf98-3905-4212-994c-447b3cae6827","Type":"ContainerDied","Data":"d1ca8bbcdeafecbe338a02441823776aa5d59d0516a005716778169430a2a815"} Oct 30 06:11:30.980648 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:30.980542 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1ca8bbcdeafecbe338a02441823776aa5d59d0516a005716778169430a2a815" Oct 30 06:11:31.463957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:31.463926 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:33.233607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:33.233577 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:34.140785 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:11:34.140746 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:11:34.950399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:34.950368 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6"] Oct 30 06:11:34.950872 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:34.950859 6999 cpu_manager.go:414] "containerMap: removing container" podUID="1cbecf98-3905-4212-994c-447b3cae6827" containerName="revoke-keys" Oct 30 06:11:34.950927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:34.950872 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cbecf98-3905-4212-994c-447b3cae6827" containerName="revoke-keys" Oct 30 06:11:34.950981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:34.950972 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cbecf98-3905-4212-994c-447b3cae6827" containerName="revoke-keys" Oct 30 06:11:34.953859 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:34.953839 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:34.987690 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:34.987663 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6"] Oct 30 06:11:35.000804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.000768 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.000943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.000839 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b253e877-6873-4e91-a50b-30fef43bcd3e-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.000943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.000870 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkmhz\" (UniqueName: \"kubernetes.io/projected/b253e877-6873-4e91-a50b-30fef43bcd3e-kube-api-access-dkmhz\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.000943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.000917 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.001045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.000948 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.001081 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.001055 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.001114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.001097 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.001184 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.001169 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.001259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.001245 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.102287 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.102241 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.102488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.102306 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.102488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.102335 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.102488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.102367 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b253e877-6873-4e91-a50b-30fef43bcd3e-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.102488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.102384 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkmhz\" (UniqueName: \"kubernetes.io/projected/b253e877-6873-4e91-a50b-30fef43bcd3e-kube-api-access-dkmhz\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.102488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.102416 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.102488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.102442 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.102823 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.102546 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.102823 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.102592 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.102823 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.102773 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.102981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.102858 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.103056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.103026 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.103180 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.103158 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b253e877-6873-4e91-a50b-30fef43bcd3e-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.103348 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.103326 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.104980 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.104957 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.105162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.105141 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.125726 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.125705 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.126125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.126103 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkmhz\" (UniqueName: \"kubernetes.io/projected/b253e877-6873-4e91-a50b-30fef43bcd3e-kube-api-access-dkmhz\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.176142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.176111 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:35.269293 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.269180 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:35.591177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.591150 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6"] Oct 30 06:11:35.592607 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:11:35.592578 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb253e877_6873_4e91_a50b_30fef43bcd3e.slice/crio-62d3228869a96ce500f57131a93759486321eb3390dc169a5db7fc90ab658746 WatchSource:0}: Error finding container 62d3228869a96ce500f57131a93759486321eb3390dc169a5db7fc90ab658746: Status 404 returned error can't find the container with id 62d3228869a96ce500f57131a93759486321eb3390dc169a5db7fc90ab658746 Oct 30 06:11:35.595052 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.595019 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:35.595118 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.595090 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:35.595152 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:35.595121 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:36.015387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:36.015353 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" event={"ID":"b253e877-6873-4e91-a50b-30fef43bcd3e","Type":"ContainerStarted","Data":"e1218689aa44472520947f883c287bc6a668ade75f5f6582cb6963408d8555a1"} Oct 30 06:11:36.015853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:36.015407 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" event={"ID":"b253e877-6873-4e91-a50b-30fef43bcd3e","Type":"ContainerStarted","Data":"62d3228869a96ce500f57131a93759486321eb3390dc169a5db7fc90ab658746"} Oct 30 06:11:36.096022 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:36.095958 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" podStartSLOduration=2.095940022 podStartE2EDuration="2.095940022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:36.093353575 +0000 UTC m=+1062.602373462" watchObservedRunningTime="2026-10-30 06:11:36.095940022 +0000 UTC m=+1062.604959915" Oct 30 06:11:36.270367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:36.270271 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:36.271787 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:36.271765 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 06:11:36.271865 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:36.271813 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" podUID="b253e877-6873-4e91-a50b-30fef43bcd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 06:11:37.065857 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:37.065822 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:37.279609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:37.279574 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 06:11:37.279875 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:37.279848 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" podUID="b253e877-6873-4e91-a50b-30fef43bcd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 06:11:38.269693 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:38.269662 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 06:11:38.270170 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:38.269725 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" podUID="b253e877-6873-4e91-a50b-30fef43bcd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 06:11:38.840235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:38.840072 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:39.270162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:39.270128 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 06:11:39.270665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:39.270187 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" podUID="b253e877-6873-4e91-a50b-30fef43bcd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 06:11:39.328695 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:39.328657 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:40.270441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.270411 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 06:11:40.270926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.270468 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" podUID="b253e877-6873-4e91-a50b-30fef43bcd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 06:11:40.662544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.662508 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n"] Oct 30 06:11:40.667480 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.667458 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.690647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.690622 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:40.698026 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.698003 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n"] Oct 30 06:11:40.773964 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.773937 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a519e9-openshift-default-dockercfg-nffns\"" Oct 30 06:11:40.801530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.801497 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495cb545-9b57-4595-88a3-3197d71ab950-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.801686 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.801556 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.801729 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.801678 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5htn\" (UniqueName: \"kubernetes.io/projected/495cb545-9b57-4595-88a3-3197d71ab950-kube-api-access-d5htn\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.801729 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.801716 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.801815 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.801779 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.801815 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.801804 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495cb545-9b57-4595-88a3-3197d71ab950-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.801903 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.801892 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.801938 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.801912 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495cb545-9b57-4595-88a3-3197d71ab950-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.801974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.801944 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.861135 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.861100 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:40.903294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.903265 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.903471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.903336 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5htn\" (UniqueName: \"kubernetes.io/projected/495cb545-9b57-4595-88a3-3197d71ab950-kube-api-access-d5htn\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.903471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.903367 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.903471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.903403 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.903471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.903425 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495cb545-9b57-4595-88a3-3197d71ab950-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.903687 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.903498 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.903687 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.903525 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495cb545-9b57-4595-88a3-3197d71ab950-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.903687 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.903574 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.903687 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.903615 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495cb545-9b57-4595-88a3-3197d71ab950-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.903916 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.903855 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.904157 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.904106 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.904310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.904156 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.904310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.904196 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495cb545-9b57-4595-88a3-3197d71ab950-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.904655 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.904628 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.905900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.905880 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.906103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.906085 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495cb545-9b57-4595-88a3-3197d71ab950-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.931700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.931635 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5htn\" (UniqueName: \"kubernetes.io/projected/495cb545-9b57-4595-88a3-3197d71ab950-kube-api-access-d5htn\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.957721 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.957689 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495cb545-9b57-4595-88a3-3197d71ab950-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:40.992964 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:40.992940 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:41.269856 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:41.269771 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 06:11:41.269856 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:41.269828 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" podUID="b253e877-6873-4e91-a50b-30fef43bcd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 06:11:41.360439 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:41.360405 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n"] Oct 30 06:11:41.365049 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:11:41.364993 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495cb545_9b57_4595_88a3_3197d71ab950.slice/crio-49a8706c289153bc6c8d46ee403eab844dea4da845fd45af3c8f750c0fb59a47 WatchSource:0}: Error finding container 49a8706c289153bc6c8d46ee403eab844dea4da845fd45af3c8f750c0fb59a47: Status 404 returned error can't find the container with id 49a8706c289153bc6c8d46ee403eab844dea4da845fd45af3c8f750c0fb59a47 Oct 30 06:11:41.367612 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:41.367575 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:41.367700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:41.367660 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:41.371237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:41.368265 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:41.577370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:41.577271 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:41.885566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:41.885535 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:42.039826 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:42.039800 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:42.081662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:42.081633 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" event={"ID":"495cb545-9b57-4595-88a3-3197d71ab950","Type":"ContainerStarted","Data":"7b366af470e311bf94c882f26d3d86de4f69618eee4367207c0ff18b38bcc1a0"} Oct 30 06:11:42.081807 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:42.081681 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" event={"ID":"495cb545-9b57-4595-88a3-3197d71ab950","Type":"ContainerStarted","Data":"49a8706c289153bc6c8d46ee403eab844dea4da845fd45af3c8f750c0fb59a47"} Oct 30 06:11:42.220060 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:42.219951 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:42.269680 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:42.269642 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 06:11:42.269886 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:42.269705 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" podUID="b253e877-6873-4e91-a50b-30fef43bcd3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 06:11:42.993689 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:42.993656 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:42.995618 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:42.995587 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 06:11:42.995801 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:42.995652 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" podUID="495cb545-9b57-4595-88a3-3197d71ab950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 06:11:43.275138 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:43.275065 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:43.275463 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:43.275438 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:43.276310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:43.276289 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:11:43.353239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:43.352521 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" podStartSLOduration=3.352501047 podStartE2EDuration="3.352501047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:42.134241744 +0000 UTC m=+1068.643261631" watchObservedRunningTime="2026-10-30 06:11:43.352501047 +0000 UTC m=+1069.861520935" Oct 30 06:11:43.687381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:43.687350 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:43.993385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:43.993301 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 06:11:43.993385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:43.993349 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" podUID="495cb545-9b57-4595-88a3-3197d71ab950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 06:11:44.996235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:44.995533 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 06:11:44.996235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:44.995593 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" podUID="495cb545-9b57-4595-88a3-3197d71ab950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 06:11:46.001238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:46.000503 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 06:11:46.001238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:46.000591 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" podUID="495cb545-9b57-4595-88a3-3197d71ab950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 06:11:46.774235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:46.773653 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:46.994103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:46.994069 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 06:11:46.994313 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:46.994120 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" podUID="495cb545-9b57-4595-88a3-3197d71ab950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 06:11:47.994314 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:47.994279 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 06:11:47.994740 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:47.994339 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" podUID="495cb545-9b57-4595-88a3-3197d71ab950" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 06:11:48.663090 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:48.663051 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:48.909643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:48.909613 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:49.182565 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.182529 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:49.360567 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.360532 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:49.407508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.407480 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:49.407906 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.407855 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:49.408902 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.408872 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:11:49.563703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.563633 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs"] Oct 30 06:11:49.567870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.567852 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.591923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.591895 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs"] Oct 30 06:11:49.621682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.621658 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.621838 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.621709 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43830d5c-1f9d-4699-a0f8-089f40e1e098-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.621838 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.621738 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.621962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.621892 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b54x\" (UniqueName: \"kubernetes.io/projected/43830d5c-1f9d-4699-a0f8-089f40e1e098-kube-api-access-2b54x\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.622123 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.622018 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.622260 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.622187 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.622346 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.622309 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.622396 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.622373 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.622456 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.622419 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.724030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.723997 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.724232 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.724067 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.724232 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.724108 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.724232 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.724152 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.724232 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.724198 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43830d5c-1f9d-4699-a0f8-089f40e1e098-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.724481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.724246 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.724481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.724289 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b54x\" (UniqueName: \"kubernetes.io/projected/43830d5c-1f9d-4699-a0f8-089f40e1e098-kube-api-access-2b54x\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.724481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.724379 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.724481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.724423 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.724681 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.724491 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.725594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.724830 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.725594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.725059 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.725594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.725141 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43830d5c-1f9d-4699-a0f8-089f40e1e098-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.725594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.725385 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.726993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.726966 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.728766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.728741 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.747576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.747548 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.747912 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.747895 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b54x\" (UniqueName: \"kubernetes.io/projected/43830d5c-1f9d-4699-a0f8-089f40e1e098-kube-api-access-2b54x\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:49.885085 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:49.885051 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:50.240128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:50.240103 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs"] Oct 30 06:11:50.241750 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:11:50.241721 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43830d5c_1f9d_4699_a0f8_089f40e1e098.slice/crio-2bc172a126c7a9f2e756f66ea6549448c80f0de0f64663df32b6974af65f04b1 WatchSource:0}: Error finding container 2bc172a126c7a9f2e756f66ea6549448c80f0de0f64663df32b6974af65f04b1: Status 404 returned error can't find the container with id 2bc172a126c7a9f2e756f66ea6549448c80f0de0f64663df32b6974af65f04b1 Oct 30 06:11:50.244340 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:50.244291 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:50.244468 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:50.244382 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:50.244468 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:50.244425 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:50.860242 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:50.858553 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:50.923878 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:50.923844 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt"] Oct 30 06:11:50.929084 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:50.929054 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" Oct 30 06:11:50.963719 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:50.962715 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:11:50.963719 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:50.962799 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:50.971754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:50.971720 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt"] Oct 30 06:11:51.065036 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.065001 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06707a7-4643-4c6a-b18e-5bcd55e16d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81bff280-nspmt\" (UID: \"e06707a7-4643-4c6a-b18e-5bcd55e16d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" Oct 30 06:11:51.065317 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.065282 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm4bt\" (UniqueName: \"kubernetes.io/projected/e06707a7-4643-4c6a-b18e-5bcd55e16d37-kube-api-access-sm4bt\") pod \"maas-api-revoke-sub-81bff280-nspmt\" (UID: \"e06707a7-4643-4c6a-b18e-5bcd55e16d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" Oct 30 06:11:51.065449 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.065431 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06707a7-4643-4c6a-b18e-5bcd55e16d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81bff280-nspmt\" (UID: \"e06707a7-4643-4c6a-b18e-5bcd55e16d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" Oct 30 06:11:51.167062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.166970 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06707a7-4643-4c6a-b18e-5bcd55e16d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81bff280-nspmt\" (UID: \"e06707a7-4643-4c6a-b18e-5bcd55e16d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" Oct 30 06:11:51.167302 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.167058 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06707a7-4643-4c6a-b18e-5bcd55e16d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81bff280-nspmt\" (UID: \"e06707a7-4643-4c6a-b18e-5bcd55e16d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" Oct 30 06:11:51.167302 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.167284 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm4bt\" (UniqueName: \"kubernetes.io/projected/e06707a7-4643-4c6a-b18e-5bcd55e16d37-kube-api-access-sm4bt\") pod \"maas-api-revoke-sub-81bff280-nspmt\" (UID: \"e06707a7-4643-4c6a-b18e-5bcd55e16d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" Oct 30 06:11:51.167861 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.167842 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06707a7-4643-4c6a-b18e-5bcd55e16d37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81bff280-nspmt\" (UID: \"e06707a7-4643-4c6a-b18e-5bcd55e16d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" Oct 30 06:11:51.169607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.169586 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06707a7-4643-4c6a-b18e-5bcd55e16d37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81bff280-nspmt\" (UID: \"e06707a7-4643-4c6a-b18e-5bcd55e16d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" Oct 30 06:11:51.190430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.190400 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm4bt\" (UniqueName: \"kubernetes.io/projected/e06707a7-4643-4c6a-b18e-5bcd55e16d37-kube-api-access-sm4bt\") pod \"maas-api-revoke-sub-81bff280-nspmt\" (UID: \"e06707a7-4643-4c6a-b18e-5bcd55e16d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" Oct 30 06:11:51.202665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.202641 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" event={"ID":"43830d5c-1f9d-4699-a0f8-089f40e1e098","Type":"ContainerStarted","Data":"89ea3ecaf3be0239493b96241d9ab807609fd98ee42d1d3a674d293680d6cf04"} Oct 30 06:11:51.202778 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.202689 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" event={"ID":"43830d5c-1f9d-4699-a0f8-089f40e1e098","Type":"ContainerStarted","Data":"2bc172a126c7a9f2e756f66ea6549448c80f0de0f64663df32b6974af65f04b1"} Oct 30 06:11:51.260090 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.251945 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" podStartSLOduration=2.251924658 podStartE2EDuration="2.251924658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:51.25054664 +0000 UTC m=+1077.759566528" watchObservedRunningTime="2026-10-30 06:11:51.251924658 +0000 UTC m=+1077.760944547" Oct 30 06:11:51.287730 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.287706 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" Oct 30 06:11:51.608544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.608516 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt"] Oct 30 06:11:51.609284 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:11:51.609258 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06707a7_4643_4c6a_b18e_5bcd55e16d37.slice/crio-f89afa72772ced20d3f85ebe4fe9ba4350e2f3fc0a1a62d8b30bd17e3d913d73 WatchSource:0}: Error finding container f89afa72772ced20d3f85ebe4fe9ba4350e2f3fc0a1a62d8b30bd17e3d913d73: Status 404 returned error can't find the container with id f89afa72772ced20d3f85ebe4fe9ba4350e2f3fc0a1a62d8b30bd17e3d913d73 Oct 30 06:11:51.886186 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.886157 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:51.887289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.887266 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 06:11:51.887413 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:51.887312 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" podUID="43830d5c-1f9d-4699-a0f8-089f40e1e098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 06:11:52.166678 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.166598 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6"] Oct 30 06:11:52.171029 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.171007 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.201634 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.201611 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6"] Oct 30 06:11:52.211466 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.211438 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" event={"ID":"e06707a7-4643-4c6a-b18e-5bcd55e16d37","Type":"ContainerStarted","Data":"2ffa56c964bc7f5b6ad7705708be16e383efaabe7475617f77f743a4fb1732ba"} Oct 30 06:11:52.211621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.211486 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" event={"ID":"e06707a7-4643-4c6a-b18e-5bcd55e16d37","Type":"ContainerStarted","Data":"f89afa72772ced20d3f85ebe4fe9ba4350e2f3fc0a1a62d8b30bd17e3d913d73"} Oct 30 06:11:52.277324 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.277278 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" podStartSLOduration=2.27726356 podStartE2EDuration="2.27726356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:52.274563887 +0000 UTC m=+1078.783583772" watchObservedRunningTime="2026-10-30 06:11:52.27726356 +0000 UTC m=+1078.786283446" Oct 30 06:11:52.279910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.279881 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.280123 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.280095 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.280228 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.280181 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.280315 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.280283 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.280393 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.280378 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.280530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.280513 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.280576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.280550 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swkt6\" (UniqueName: \"kubernetes.io/projected/d9e281b2-b9c9-47a9-bb0a-96c313f43554-kube-api-access-swkt6\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.280629 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.280577 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.280629 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.280611 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.382263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.382203 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.382421 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.382280 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.382421 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.382320 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.382421 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.382363 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.382573 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.382444 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.382573 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.382471 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swkt6\" (UniqueName: \"kubernetes.io/projected/d9e281b2-b9c9-47a9-bb0a-96c313f43554-kube-api-access-swkt6\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.382672 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.382611 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.382672 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.382657 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.382766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.382685 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.382766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.382743 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.382766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.382750 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.382983 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.382959 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.383055 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.383038 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.383102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.383042 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.385022 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.385001 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.385241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.385203 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.405312 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.405282 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.408582 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.408557 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swkt6\" (UniqueName: \"kubernetes.io/projected/d9e281b2-b9c9-47a9-bb0a-96c313f43554-kube-api-access-swkt6\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.485906 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.485826 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:52.865356 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.859740 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6"] Oct 30 06:11:52.869542 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:11:52.869513 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e281b2_b9c9_47a9_bb0a_96c313f43554.slice/crio-05ee5d560cd4498443a22b3da7fddd880d8c5e7d7f0eaf3ad6ec32fa8e0c305f WatchSource:0}: Error finding container 05ee5d560cd4498443a22b3da7fddd880d8c5e7d7f0eaf3ad6ec32fa8e0c305f: Status 404 returned error can't find the container with id 05ee5d560cd4498443a22b3da7fddd880d8c5e7d7f0eaf3ad6ec32fa8e0c305f Oct 30 06:11:52.872369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.872329 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:52.872460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.872418 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:52.872521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.872457 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:52.885746 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.885728 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 06:11:52.885811 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.885771 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" podUID="43830d5c-1f9d-4699-a0f8-089f40e1e098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 06:11:52.921843 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:52.921821 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:53.219307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:53.219256 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" event={"ID":"d9e281b2-b9c9-47a9-bb0a-96c313f43554","Type":"ContainerStarted","Data":"fddab5b5250d520413f9efc68dc7a12fa56282337a72792d4be614979d701647"} Oct 30 06:11:53.219459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:53.219331 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" event={"ID":"d9e281b2-b9c9-47a9-bb0a-96c313f43554","Type":"ContainerStarted","Data":"05ee5d560cd4498443a22b3da7fddd880d8c5e7d7f0eaf3ad6ec32fa8e0c305f"} Oct 30 06:11:53.220986 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:53.220962 6999 generic.go:356] "Generic (PLEG): container finished" podID="e06707a7-4643-4c6a-b18e-5bcd55e16d37" containerID="2ffa56c964bc7f5b6ad7705708be16e383efaabe7475617f77f743a4fb1732ba" exitCode=0 Oct 30 06:11:53.221112 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:53.221025 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" event={"ID":"e06707a7-4643-4c6a-b18e-5bcd55e16d37","Type":"ContainerDied","Data":"2ffa56c964bc7f5b6ad7705708be16e383efaabe7475617f77f743a4fb1732ba"} Oct 30 06:11:53.302817 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:53.302786 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:53.324011 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:53.323960 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" podStartSLOduration=1.323946482 podStartE2EDuration="1.323946482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:53.268473359 +0000 UTC m=+1079.777493246" watchObservedRunningTime="2026-10-30 06:11:53.323946482 +0000 UTC m=+1079.832966368" Oct 30 06:11:53.486312 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:53.486203 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:11:53.487703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:53.487678 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 06:11:53.487849 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:53.487732 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" podUID="d9e281b2-b9c9-47a9-bb0a-96c313f43554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 06:11:53.512742 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:53.508826 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:53.892242 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:53.891890 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 06:11:53.892242 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:53.891950 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" podUID="43830d5c-1f9d-4699-a0f8-089f40e1e098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 06:11:54.218361 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:54.218291 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:54.505072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:54.504780 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 06:11:54.505072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:54.504838 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" podUID="d9e281b2-b9c9-47a9-bb0a-96c313f43554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 06:11:54.532779 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:54.532756 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" Oct 30 06:11:54.623620 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:54.623583 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06707a7-4643-4c6a-b18e-5bcd55e16d37-kube-api-access-sm4bt\" (UniqueName: \"kubernetes.io/projected/e06707a7-4643-4c6a-b18e-5bcd55e16d37-kube-api-access-sm4bt\") pod \"e06707a7-4643-4c6a-b18e-5bcd55e16d37\" (UID: \"e06707a7-4643-4c6a-b18e-5bcd55e16d37\") " Oct 30 06:11:54.623813 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:54.623691 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e06707a7-4643-4c6a-b18e-5bcd55e16d37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06707a7-4643-4c6a-b18e-5bcd55e16d37-maas-api-xks-ca\") pod \"e06707a7-4643-4c6a-b18e-5bcd55e16d37\" (UID: \"e06707a7-4643-4c6a-b18e-5bcd55e16d37\") " Oct 30 06:11:54.623813 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:54.623713 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e06707a7-4643-4c6a-b18e-5bcd55e16d37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06707a7-4643-4c6a-b18e-5bcd55e16d37-maas-api-openshift-ca\") pod \"e06707a7-4643-4c6a-b18e-5bcd55e16d37\" (UID: \"e06707a7-4643-4c6a-b18e-5bcd55e16d37\") " Oct 30 06:11:54.624453 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:54.624430 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06707a7-4643-4c6a-b18e-5bcd55e16d37-maas-api-openshift-ca" pod "e06707a7-4643-4c6a-b18e-5bcd55e16d37" (UID: "e06707a7-4643-4c6a-b18e-5bcd55e16d37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:54.625752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:54.625727 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06707a7-4643-4c6a-b18e-5bcd55e16d37-kube-api-access-sm4bt" pod "e06707a7-4643-4c6a-b18e-5bcd55e16d37" (UID: "e06707a7-4643-4c6a-b18e-5bcd55e16d37"). InnerVolumeSpecName "kube-api-access-sm4bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:54.626702 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:54.626677 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e06707a7-4643-4c6a-b18e-5bcd55e16d37-maas-api-xks-ca" pod "e06707a7-4643-4c6a-b18e-5bcd55e16d37" (UID: "e06707a7-4643-4c6a-b18e-5bcd55e16d37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:54.725410 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:54.725385 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06707a7-4643-4c6a-b18e-5bcd55e16d37-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:11:54.725601 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:54.725590 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06707a7-4643-4c6a-b18e-5bcd55e16d37-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:11:54.725658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:54.725650 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm4bt\" (UniqueName: \"kubernetes.io/projected/e06707a7-4643-4c6a-b18e-5bcd55e16d37-kube-api-access-sm4bt\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:11:54.886236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:54.885605 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 06:11:54.886236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:54.885664 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" podUID="43830d5c-1f9d-4699-a0f8-089f40e1e098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 06:11:55.261298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:55.246497 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:55.294364 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:55.294334 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" Oct 30 06:11:55.294559 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:55.294382 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt" event={"ID":"e06707a7-4643-4c6a-b18e-5bcd55e16d37","Type":"ContainerDied","Data":"f89afa72772ced20d3f85ebe4fe9ba4350e2f3fc0a1a62d8b30bd17e3d913d73"} Oct 30 06:11:55.294559 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:55.294479 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f89afa72772ced20d3f85ebe4fe9ba4350e2f3fc0a1a62d8b30bd17e3d913d73" Oct 30 06:11:55.398048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:55.397973 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:55.494309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:55.487630 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 06:11:55.494309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:55.487690 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" podUID="d9e281b2-b9c9-47a9-bb0a-96c313f43554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 06:11:55.887254 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:55.886564 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 06:11:55.887254 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:55.886617 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" podUID="43830d5c-1f9d-4699-a0f8-089f40e1e098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 06:11:56.486533 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:56.486500 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 06:11:56.486731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:56.486561 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" podUID="d9e281b2-b9c9-47a9-bb0a-96c313f43554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 06:11:56.885461 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:56.885430 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 06:11:56.885632 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:56.885486 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" podUID="43830d5c-1f9d-4699-a0f8-089f40e1e098" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 06:11:57.382809 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:57.382776 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:57.487195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:57.486736 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 06:11:57.487195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:57.486794 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" podUID="d9e281b2-b9c9-47a9-bb0a-96c313f43554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 06:11:57.768627 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:57.768533 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:57.900616 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:57.900584 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:57.901465 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:57.901437 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:57.901916 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:57.901902 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:11:58.486142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:58.486106 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 06:11:58.486657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:58.486174 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" podUID="d9e281b2-b9c9-47a9-bb0a-96c313f43554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 06:11:58.567683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:58.567658 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:59.348894 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:59.348848 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:11:59.486515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:59.486487 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 06:11:59.486959 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:59.486548 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" podUID="d9e281b2-b9c9-47a9-bb0a-96c313f43554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 06:11:59.490560 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:11:59.490539 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:00.036264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.036234 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs"] Oct 30 06:12:00.036807 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.036780 6999 cpu_manager.go:414] "containerMap: removing container" podUID="e06707a7-4643-4c6a-b18e-5bcd55e16d37" containerName="revoke-keys" Oct 30 06:12:00.036879 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.036810 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06707a7-4643-4c6a-b18e-5bcd55e16d37" containerName="revoke-keys" Oct 30 06:12:00.036995 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.036984 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06707a7-4643-4c6a-b18e-5bcd55e16d37" containerName="revoke-keys" Oct 30 06:12:00.040261 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.040237 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" Oct 30 06:12:00.060145 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.060118 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs"] Oct 30 06:12:00.121192 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.121158 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm6t2\" (UniqueName: \"kubernetes.io/projected/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-kube-api-access-sm6t2\") pod \"maas-api-revoke-sub-9bb09bcb-cmnfs\" (UID: \"15ea8ec2-12f7-491b-97ba-deedcd92f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" Oct 30 06:12:00.121415 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.121389 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bb09bcb-cmnfs\" (UID: \"15ea8ec2-12f7-491b-97ba-deedcd92f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" Oct 30 06:12:00.121489 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.121445 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bb09bcb-cmnfs\" (UID: \"15ea8ec2-12f7-491b-97ba-deedcd92f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" Oct 30 06:12:00.136635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.136611 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:12:00.136750 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.136700 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:12:00.222861 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.222828 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bb09bcb-cmnfs\" (UID: \"15ea8ec2-12f7-491b-97ba-deedcd92f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" Oct 30 06:12:00.223014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.222876 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bb09bcb-cmnfs\" (UID: \"15ea8ec2-12f7-491b-97ba-deedcd92f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" Oct 30 06:12:00.223014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.222986 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6t2\" (UniqueName: \"kubernetes.io/projected/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-kube-api-access-sm6t2\") pod \"maas-api-revoke-sub-9bb09bcb-cmnfs\" (UID: \"15ea8ec2-12f7-491b-97ba-deedcd92f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" Oct 30 06:12:00.223590 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.223565 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bb09bcb-cmnfs\" (UID: \"15ea8ec2-12f7-491b-97ba-deedcd92f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" Oct 30 06:12:00.225514 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.225491 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bb09bcb-cmnfs\" (UID: \"15ea8ec2-12f7-491b-97ba-deedcd92f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" Oct 30 06:12:00.246085 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.246064 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm6t2\" (UniqueName: \"kubernetes.io/projected/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-kube-api-access-sm6t2\") pod \"maas-api-revoke-sub-9bb09bcb-cmnfs\" (UID: \"15ea8ec2-12f7-491b-97ba-deedcd92f3ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" Oct 30 06:12:00.356894 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.356826 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" Oct 30 06:12:00.448202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.448052 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:00.490791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.490765 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:12:00.491164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.491120 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:12:00.491841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.491823 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:12:00.659725 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:00.659705 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs"] Oct 30 06:12:00.661438 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:12:00.661414 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ea8ec2_12f7_491b_97ba_deedcd92f3ef.slice/crio-227e06ef847a3b7fd2f10a9aa975b135e31b5dd270fca743405afe57df7ea9b5 WatchSource:0}: Error finding container 227e06ef847a3b7fd2f10a9aa975b135e31b5dd270fca743405afe57df7ea9b5: Status 404 returned error can't find the container with id 227e06ef847a3b7fd2f10a9aa975b135e31b5dd270fca743405afe57df7ea9b5 Oct 30 06:12:01.037342 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:01.037312 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:01.341951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:01.341864 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" event={"ID":"15ea8ec2-12f7-491b-97ba-deedcd92f3ef","Type":"ContainerStarted","Data":"81a2428d1251b7f6c87b393f447b1e67e780b74d697720011e7933e9c9ae83c9"} Oct 30 06:12:01.341951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:01.341950 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" event={"ID":"15ea8ec2-12f7-491b-97ba-deedcd92f3ef","Type":"ContainerStarted","Data":"227e06ef847a3b7fd2f10a9aa975b135e31b5dd270fca743405afe57df7ea9b5"} Oct 30 06:12:01.372295 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:01.372260 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:01.383635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:01.383594 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" podStartSLOduration=1.383580937 podStartE2EDuration="1.383580937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:01.381223526 +0000 UTC m=+1087.890243405" watchObservedRunningTime="2026-10-30 06:12:01.383580937 +0000 UTC m=+1087.892600824" Oct 30 06:12:02.354288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:02.354247 6999 generic.go:356] "Generic (PLEG): container finished" podID="15ea8ec2-12f7-491b-97ba-deedcd92f3ef" containerID="81a2428d1251b7f6c87b393f447b1e67e780b74d697720011e7933e9c9ae83c9" exitCode=0 Oct 30 06:12:02.354820 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:02.354426 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" event={"ID":"15ea8ec2-12f7-491b-97ba-deedcd92f3ef","Type":"ContainerDied","Data":"81a2428d1251b7f6c87b393f447b1e67e780b74d697720011e7933e9c9ae83c9"} Oct 30 06:12:03.160237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:03.159103 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:04.438340 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.438296 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" podUID="776861d0-f966-44ca-b0eb-22b1a1b47e79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:04.476420 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.476372 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" podUID="79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:04.586439 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.586406 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" Oct 30 06:12:04.655290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.633496 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-maas-api-openshift-ca\") pod \"15ea8ec2-12f7-491b-97ba-deedcd92f3ef\" (UID: \"15ea8ec2-12f7-491b-97ba-deedcd92f3ef\") " Oct 30 06:12:04.685381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.685349 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-kube-api-access-sm6t2\" (UniqueName: \"kubernetes.io/projected/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-kube-api-access-sm6t2\") pod \"15ea8ec2-12f7-491b-97ba-deedcd92f3ef\" (UID: \"15ea8ec2-12f7-491b-97ba-deedcd92f3ef\") " Oct 30 06:12:04.685754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.685730 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-maas-api-xks-ca\") pod \"15ea8ec2-12f7-491b-97ba-deedcd92f3ef\" (UID: \"15ea8ec2-12f7-491b-97ba-deedcd92f3ef\") " Oct 30 06:12:04.686955 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.659618 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-maas-api-openshift-ca" pod "15ea8ec2-12f7-491b-97ba-deedcd92f3ef" (UID: "15ea8ec2-12f7-491b-97ba-deedcd92f3ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:12:04.687072 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:12:04.657102 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:12:04.728975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.712777 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" podUID="b2701c51-6377-411c-9924-715ff0765564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:04.728975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.713420 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7"] Oct 30 06:12:04.728975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.714132 6999 cpu_manager.go:414] "containerMap: removing container" podUID="15ea8ec2-12f7-491b-97ba-deedcd92f3ef" containerName="revoke-keys" Oct 30 06:12:04.728975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.714145 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ea8ec2-12f7-491b-97ba-deedcd92f3ef" containerName="revoke-keys" Oct 30 06:12:04.728975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.714344 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ea8ec2-12f7-491b-97ba-deedcd92f3ef" containerName="revoke-keys" Oct 30 06:12:04.739255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.736609 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:04.870179 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.870139 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:04.947255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.946813 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7"] Oct 30 06:12:04.953237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.948251 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:04.953237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.948334 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:04.953237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.948383 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:04.953237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.948414 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:04.953237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.948488 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:04.953237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.948558 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:04.953237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.948611 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8frvh\" (UniqueName: \"kubernetes.io/projected/6109420f-b8c7-4f71-8796-5e79cdcd53b0-kube-api-access-8frvh\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:04.953237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.948720 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:04.953237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:04.948824 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:12:05.042306 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.041635 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-kube-api-access-sm6t2" pod "15ea8ec2-12f7-491b-97ba-deedcd92f3ef" (UID: "15ea8ec2-12f7-491b-97ba-deedcd92f3ef"). InnerVolumeSpecName "kube-api-access-sm6t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:05.050978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.050944 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8frvh\" (UniqueName: \"kubernetes.io/projected/6109420f-b8c7-4f71-8796-5e79cdcd53b0-kube-api-access-8frvh\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.051309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.051290 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.051512 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.051494 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.051651 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.051632 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.051771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.051757 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.051895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.051878 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.052012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.051997 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.052172 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.052154 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.052388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.052371 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.052591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.052577 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm6t2\" (UniqueName: \"kubernetes.io/projected/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-kube-api-access-sm6t2\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:12:05.058658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.058635 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.059976 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.059658 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.060405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.060385 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.060764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.060745 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.061138 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.061118 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.072984 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.072355 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-maas-api-xks-ca" pod "15ea8ec2-12f7-491b-97ba-deedcd92f3ef" (UID: "15ea8ec2-12f7-491b-97ba-deedcd92f3ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:12:05.099285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.099099 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8frvh\" (UniqueName: \"kubernetes.io/projected/6109420f-b8c7-4f71-8796-5e79cdcd53b0-kube-api-access-8frvh\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.100441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.099815 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.100441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.100369 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.104717 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.104669 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.154185 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.154135 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15ea8ec2-12f7-491b-97ba-deedcd92f3ef-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:12:05.249151 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.247764 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:05.285972 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.285641 6999 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" podUID="8e38cac7-3426-497c-8b84-e7552169bb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:05.537238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.513706 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" podUID="8e38cac7-3426-497c-8b84-e7552169bb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:05.584304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.580757 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" podUID="79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 06:12:05.584304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.582249 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:12:05.725710 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:05.725675 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" Oct 30 06:12:06.180241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.156622 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs" event={"ID":"15ea8ec2-12f7-491b-97ba-deedcd92f3ef","Type":"ContainerDied","Data":"227e06ef847a3b7fd2f10a9aa975b135e31b5dd270fca743405afe57df7ea9b5"} Oct 30 06:12:06.209313 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.209279 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="227e06ef847a3b7fd2f10a9aa975b135e31b5dd270fca743405afe57df7ea9b5" Oct 30 06:12:06.271459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.271425 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz"] Oct 30 06:12:06.286397 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.286345 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" podUID="5990b367-6bd6-483b-ab76-35375b015cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:06.395537 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.395507 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:06.425912 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.425878 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b4k9b\"" Oct 30 06:12:06.426110 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.425932 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ba44b248-serving-cert\"" Oct 30 06:12:06.426259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.426064 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 06:12:06.482241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.452310 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz"] Oct 30 06:12:06.493243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.493028 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpb4\" (UniqueName: \"kubernetes.io/projected/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-kube-api-access-jmpb4\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:06.493243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.493144 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-maas-api-tls\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:06.493243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.493185 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-openshift-service-ca\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:06.522238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.507400 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-metrics-certs\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:06.603256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.602801 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" podUID="8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:06.603256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.603179 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" podUID="79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 06:12:06.664296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.616060 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpb4\" (UniqueName: \"kubernetes.io/projected/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-kube-api-access-jmpb4\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:06.664296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.621436 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-maas-api-tls\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:06.664296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.621491 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-openshift-service-ca\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:06.664296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.621568 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-metrics-certs\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:06.664639 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.664318 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" podUID="776861d0-f966-44ca-b0eb-22b1a1b47e79" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:06.702238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.677297 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" podUID="b2701c51-6377-411c-9924-715ff0765564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:06.702238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.686877 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-openshift-service-ca\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:06.785235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:06.785058 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:07.232406 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.232364 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" podUID="987180e9-4669-4740-b2df-effebab90159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:07.316688 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.312996 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-metrics-certs\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:07.320236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.318614 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpb4\" (UniqueName: \"kubernetes.io/projected/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-kube-api-access-jmpb4\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:07.320236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.318679 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7"] Oct 30 06:12:07.320236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.318765 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:12:07.320236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.319008 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" podUID="5990b367-6bd6-483b-ab76-35375b015cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 06:12:07.320544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.320386 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:12:07.396239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.392800 6999 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" podUID="987180e9-4669-4740-b2df-effebab90159" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:07.529030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.528999 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-maas-api-tls\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:07.533963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.533934 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:07.541425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.541397 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" podUID="8e38cac7-3426-497c-8b84-e7552169bb7c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:07.543895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.543860 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:07.547385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.547359 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:07.547570 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.547548 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:07.711419 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.709444 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:07.711419 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.711379 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" podUID="8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:12:07.712023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:07.711995 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:12:08.052240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:08.042989 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 06:12:08.136685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:08.136663 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 06:12:08.727662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:08.714867 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz"] Oct 30 06:12:08.756521 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:12:08.756485 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55bbb9cb_2f0f_4a8d_ab25_e707a48d0f81.slice/crio-acd564beb035e5cf3de02b9657258723f97920db5e2fec69b27d54cd5f9d4228 WatchSource:0}: Error finding container acd564beb035e5cf3de02b9657258723f97920db5e2fec69b27d54cd5f9d4228: Status 404 returned error can't find the container with id acd564beb035e5cf3de02b9657258723f97920db5e2fec69b27d54cd5f9d4228 Oct 30 06:12:08.805498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:08.805462 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" event={"ID":"6109420f-b8c7-4f71-8796-5e79cdcd53b0","Type":"ContainerStarted","Data":"ea28ad044f2c7e24f53b338f5049b871be3017b209c0767c5967f47f392465b5"} Oct 30 06:12:08.805781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:08.805755 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" event={"ID":"6109420f-b8c7-4f71-8796-5e79cdcd53b0","Type":"ContainerStarted","Data":"7698a77ca02d2e4d45e809256cfe662461af28f5b22a0d47fe69ffce2034308c"} Oct 30 06:12:08.815244 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:08.815176 6999 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:12:08.868459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:08.866160 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" podStartSLOduration=4.866138685 podStartE2EDuration="4.866138685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:08.856744302 +0000 UTC m=+1095.365764188" watchObservedRunningTime="2026-10-30 06:12:08.866138685 +0000 UTC m=+1095.375158572" Oct 30 06:12:09.249238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:09.248490 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:09.255255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:09.250781 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 06:12:09.255255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:09.250836 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" podUID="6109420f-b8c7-4f71-8796-5e79cdcd53b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 06:12:09.605923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:09.605837 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:09.802089 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:09.802048 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" event={"ID":"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81","Type":"ContainerStarted","Data":"acd564beb035e5cf3de02b9657258723f97920db5e2fec69b27d54cd5f9d4228"} Oct 30 06:12:10.259282 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:10.258527 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 06:12:10.259282 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:10.258598 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" podUID="6109420f-b8c7-4f71-8796-5e79cdcd53b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 06:12:11.087136 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.087062 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn"] Oct 30 06:12:11.091315 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.091295 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.166588 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.166408 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn"] Oct 30 06:12:11.190861 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.190828 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knlpk\" (UniqueName: \"kubernetes.io/projected/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-kube-api-access-knlpk\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.191012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.190878 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.191012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.190904 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.191012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.190968 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.191012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.190995 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.191161 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.191042 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.191203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.191175 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.191633 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.191257 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.191633 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.191292 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.254239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.248568 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 06:12:11.254239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.248641 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" podUID="6109420f-b8c7-4f71-8796-5e79cdcd53b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 06:12:11.292166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.292130 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.292381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.292276 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.292381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.292329 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.292513 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.292388 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knlpk\" (UniqueName: \"kubernetes.io/projected/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-kube-api-access-knlpk\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.292513 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.292423 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.292513 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.292440 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.292513 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.292460 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.292513 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.292487 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.292769 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.292561 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.292769 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.292628 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.292769 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.292745 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.292992 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.292971 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.293081 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.293028 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.293139 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.293112 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.294844 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.294824 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.295155 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.295137 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.315758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.315733 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.315957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.315936 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knlpk\" (UniqueName: \"kubernetes.io/projected/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-kube-api-access-knlpk\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rrwcn\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.430362 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.430332 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:11.632260 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.631950 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:11.799041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.799013 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn"] Oct 30 06:12:11.800357 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:12:11.800332 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80eaf9bc_b46e_4bf7_a81e_f6f55b9994ca.slice/crio-3ea5b1c2995e19c82305f7d0c8e3974eff74eae832223f41a77090d860fd54fd WatchSource:0}: Error finding container 3ea5b1c2995e19c82305f7d0c8e3974eff74eae832223f41a77090d860fd54fd: Status 404 returned error can't find the container with id 3ea5b1c2995e19c82305f7d0c8e3974eff74eae832223f41a77090d860fd54fd Oct 30 06:12:11.802758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.802728 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:11.802821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.802793 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:11.802853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.802822 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:11.815813 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.815789 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" event={"ID":"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca","Type":"ContainerStarted","Data":"3ea5b1c2995e19c82305f7d0c8e3974eff74eae832223f41a77090d860fd54fd"} Oct 30 06:12:11.817333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.817312 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" event={"ID":"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81","Type":"ContainerStarted","Data":"5c68d1d1578cf05d6ce235797e167e6ead04a5ca68013eeef3cb0983b984db81"} Oct 30 06:12:11.817601 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.817582 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:11.866903 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.866845 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" podStartSLOduration=3.917785652 podStartE2EDuration="5.866829189s" totalImagesPullingTime="1.949043537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:11.864706526 +0000 UTC m=+1098.373726409" watchObservedRunningTime="2026-10-30 06:12:11.866829189 +0000 UTC m=+1098.375849078" Oct 30 06:12:11.896285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:11.896262 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:12.249037 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:12.248583 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 06:12:12.249037 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:12.248655 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" podUID="6109420f-b8c7-4f71-8796-5e79cdcd53b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 06:12:12.401459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:12.401419 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:12.658238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:12.655127 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:12.827357 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:12.827315 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" event={"ID":"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca","Type":"ContainerStarted","Data":"e13604ae1476fbf3addca593a6f6408c02453e42175f3966a123e4199772b0bd"} Oct 30 06:12:13.261240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:13.249563 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 06:12:13.261240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:13.249617 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" podUID="6109420f-b8c7-4f71-8796-5e79cdcd53b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 06:12:13.432863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:13.432580 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:13.434247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:13.434185 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rrwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 06:12:13.434405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:13.434274 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" podUID="80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 06:12:13.929673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:13.929635 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:14.248648 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:14.248546 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 06:12:14.248884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:14.248848 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" podUID="6109420f-b8c7-4f71-8796-5e79cdcd53b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 06:12:14.431133 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:14.431099 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rrwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 06:12:14.431602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:14.431155 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" podUID="80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 06:12:15.257243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:15.254948 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 06:12:15.257243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:15.255006 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" podUID="6109420f-b8c7-4f71-8796-5e79cdcd53b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 06:12:15.430673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:15.430645 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rrwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 06:12:15.430825 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:15.430765 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" podUID="80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 06:12:16.230948 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:16.230903 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:16.401858 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:16.401789 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:16.413125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:16.409982 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:16.413125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:16.411649 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:16.430823 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:16.430797 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:12:16.433429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:16.433401 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rrwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 06:12:16.433666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:16.433641 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" podUID="80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 06:12:16.493803 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:16.493642 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" podStartSLOduration=5.493589828 podStartE2EDuration="5.493589828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:12.896499715 +0000 UTC m=+1099.405519613" watchObservedRunningTime="2026-10-30 06:12:16.493589828 +0000 UTC m=+1103.002609757" Oct 30 06:12:17.437622 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:17.437589 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rrwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 06:12:17.438205 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:17.438175 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" podUID="80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 06:12:17.851546 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:17.851460 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:12:18.434367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:18.431330 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rrwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 06:12:18.434367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:18.431388 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" podUID="80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 06:12:18.483620 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:18.483585 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:18.661390 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:18.661343 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:18.860974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:18.858406 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:19.434985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:19.434950 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rrwcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 06:12:19.435344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:19.435299 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" podUID="80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 06:12:20.843909 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:20.843858 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:21.250079 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:21.250045 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:21.251645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:21.251621 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:21.253936 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:21.253912 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:12:22.633900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:22.633868 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:24.784142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:24.784105 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:27.285434 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:27.285401 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:28.693803 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:28.693772 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:29.437541 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:29.437510 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:30.675431 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:30.675402 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:31.231521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.231489 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf"] Oct 30 06:12:31.237715 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.237684 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:31.256958 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.256930 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf"] Oct 30 06:12:31.264861 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.264827 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-1ec766-serving-cert\"" Oct 30 06:12:31.355438 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.355397 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e8576b-5ebe-47fa-aeb5-d63701c9018d-openshift-service-ca\") pod \"maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:31.355621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.355448 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfbbg\" (UniqueName: \"kubernetes.io/projected/78e8576b-5ebe-47fa-aeb5-d63701c9018d-kube-api-access-cfbbg\") pod \"maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:31.355621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.355597 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78e8576b-5ebe-47fa-aeb5-d63701c9018d-maas-api-tls\") pod \"maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:31.355695 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.355627 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e8576b-5ebe-47fa-aeb5-d63701c9018d-metrics-certs\") pod \"maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:31.356417 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.356390 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:31.457096 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.457064 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e8576b-5ebe-47fa-aeb5-d63701c9018d-openshift-service-ca\") pod \"maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:31.457096 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.457100 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfbbg\" (UniqueName: \"kubernetes.io/projected/78e8576b-5ebe-47fa-aeb5-d63701c9018d-kube-api-access-cfbbg\") pod \"maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:31.457394 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.457133 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78e8576b-5ebe-47fa-aeb5-d63701c9018d-maas-api-tls\") pod \"maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:31.457394 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.457155 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e8576b-5ebe-47fa-aeb5-d63701c9018d-metrics-certs\") pod \"maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:31.457911 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.457872 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e8576b-5ebe-47fa-aeb5-d63701c9018d-openshift-service-ca\") pod \"maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:31.459694 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.459674 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78e8576b-5ebe-47fa-aeb5-d63701c9018d-maas-api-tls\") pod \"maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:31.459804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.459780 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e8576b-5ebe-47fa-aeb5-d63701c9018d-metrics-certs\") pod \"maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:31.481607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.481508 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfbbg\" (UniqueName: \"kubernetes.io/projected/78e8576b-5ebe-47fa-aeb5-d63701c9018d-kube-api-access-cfbbg\") pod \"maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:31.505925 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.505768 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:31.551739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.551712 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:31.867388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:31.867356 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf"] Oct 30 06:12:31.868389 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:12:31.868357 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e8576b_5ebe_47fa_aeb5_d63701c9018d.slice/crio-92410a6788dd695e8be2eaf5b9dcec73747f49797b624af9d8308f48c9275bfe WatchSource:0}: Error finding container 92410a6788dd695e8be2eaf5b9dcec73747f49797b624af9d8308f48c9275bfe: Status 404 returned error can't find the container with id 92410a6788dd695e8be2eaf5b9dcec73747f49797b624af9d8308f48c9275bfe Oct 30 06:12:32.086962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:32.086930 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" event={"ID":"78e8576b-5ebe-47fa-aeb5-d63701c9018d","Type":"ContainerStarted","Data":"92410a6788dd695e8be2eaf5b9dcec73747f49797b624af9d8308f48c9275bfe"} Oct 30 06:12:33.095358 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:33.095320 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" event={"ID":"78e8576b-5ebe-47fa-aeb5-d63701c9018d","Type":"ContainerStarted","Data":"05a5dce0e31c24d9df55d4f70683bfd9d80a3686ff5884242ecf68dd2957a876"} Oct 30 06:12:33.095837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:33.095411 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:33.139233 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:33.139158 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" podStartSLOduration=1.798372835 podStartE2EDuration="2.139145619s" totalImagesPullingTime="340.772784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:33.138228002 +0000 UTC m=+1119.647247884" watchObservedRunningTime="2026-10-30 06:12:33.139145619 +0000 UTC m=+1119.648165506" Oct 30 06:12:33.221350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:33.221321 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:34.143408 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:12:34.143357 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:12:35.073555 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:35.073516 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:36.842056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:36.842017 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:38.791976 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:38.791944 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:39.108088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:39.108018 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:12:40.614920 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:40.614884 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:40.850185 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:40.850149 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk"] Oct 30 06:12:40.854112 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:40.854097 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:40.892133 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:40.892057 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk"] Oct 30 06:12:40.911088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:40.911034 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:40.911088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:40.911083 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:40.911453 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:40.911114 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:40.911453 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:40.911163 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:40.911453 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:40.911389 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:40.911453 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:40.911436 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:40.911674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:40.911529 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:40.911674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:40.911555 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:40.911674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:40.911623 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjc2w\" (UniqueName: \"kubernetes.io/projected/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-kube-api-access-sjc2w\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.012496 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.012458 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.012689 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.012503 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.012689 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.012528 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.012689 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.012633 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.012689 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.012659 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.012899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.012735 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjc2w\" (UniqueName: \"kubernetes.io/projected/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-kube-api-access-sjc2w\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.012899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.012879 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.013007 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.012900 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.013007 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.012911 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.013007 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.012973 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.013273 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.012931 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.013273 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.013205 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.013413 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.013324 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.013413 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.013348 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.015030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.015000 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.015415 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.015398 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.035919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.035894 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.036593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.036575 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjc2w\" (UniqueName: \"kubernetes.io/projected/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-kube-api-access-sjc2w\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.170654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.170556 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:41.503546 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.503522 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk"] Oct 30 06:12:41.504668 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:12:41.504646 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30fe423_3a43_4f07_aa4a_f147ea8f7cd9.slice/crio-c02a9a3776876b94238dcc80ed15ca8b0432b40ceb30209f619c27f0b123d38f WatchSource:0}: Error finding container c02a9a3776876b94238dcc80ed15ca8b0432b40ceb30209f619c27f0b123d38f: Status 404 returned error can't find the container with id c02a9a3776876b94238dcc80ed15ca8b0432b40ceb30209f619c27f0b123d38f Oct 30 06:12:41.507029 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.506992 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:41.507131 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.507070 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:41.507131 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:41.507115 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:42.170775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:42.170741 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" event={"ID":"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9","Type":"ContainerStarted","Data":"0c7a17ce0ce13651dcbca2e4787d9df892813081fb7b7d5e0c75588f17417cbf"} Oct 30 06:12:42.171246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:42.170830 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" event={"ID":"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9","Type":"ContainerStarted","Data":"c02a9a3776876b94238dcc80ed15ca8b0432b40ceb30209f619c27f0b123d38f"} Oct 30 06:12:42.460093 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:42.460009 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:43.171036 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:43.171005 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:43.172318 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:43.172291 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 06:12:43.172449 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:43.172349 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" podUID="b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 06:12:44.171454 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:44.171420 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 06:12:44.171852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:44.171485 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" podUID="b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 06:12:44.567938 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:44.567841 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:45.172374 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:45.172336 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 06:12:45.172873 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:45.172403 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" podUID="b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 06:12:46.170874 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:46.170842 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 06:12:46.171138 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:46.170897 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" podUID="b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 06:12:46.458650 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:46.458548 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:47.171319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:47.171273 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 06:12:47.171545 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:47.171354 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" podUID="b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 06:12:48.171645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:48.171607 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 06:12:48.172128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:48.171672 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" podUID="b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 06:12:48.284078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:48.284029 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:49.175605 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:49.175574 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:49.176086 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:49.175992 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:49.176677 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:49.176657 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:12:49.235706 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:49.235655 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" podStartSLOduration=9.235640441 podStartE2EDuration="9.235640441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:42.245144678 +0000 UTC m=+1128.754164565" watchObservedRunningTime="2026-10-30 06:12:49.235640441 +0000 UTC m=+1135.744660390" Oct 30 06:12:50.335373 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:50.335335 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:51.675365 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:51.675328 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:52.288036 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.288001 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:52.386329 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.386294 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq"] Oct 30 06:12:52.392735 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.392711 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.447123 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.447084 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq"] Oct 30 06:12:52.465146 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.465116 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:52.492487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.492452 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.492811 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.492782 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.492942 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.492826 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.493021 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.492997 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.493151 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.493128 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.493279 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.493252 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.493370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.493292 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.493370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.493362 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjxvz\" (UniqueName: \"kubernetes.io/projected/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-kube-api-access-vjxvz\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.493471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.493437 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.594591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.594499 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.594591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.594535 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.594591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.594572 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.594882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.594702 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.594882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.594833 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.595003 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.594911 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.595003 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.594937 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-credential-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.595122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.595020 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.595122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.595049 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.595122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.595021 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-workload-socket\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.595122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.595111 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjxvz\" (UniqueName: \"kubernetes.io/projected/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-kube-api-access-vjxvz\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.595385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.595272 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-data\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.595385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.595336 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-workload-certs\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.595712 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.595691 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istiod-ca-cert\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.597161 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.597143 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-envoy\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.597510 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.597490 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-podinfo\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.618591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.618558 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjxvz\" (UniqueName: \"kubernetes.io/projected/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-kube-api-access-vjxvz\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.618591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.618584 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-token\") pod \"e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:52.726744 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:52.726714 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:53.068731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:53.068699 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq"] Oct 30 06:12:53.070653 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:12:53.070627 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d7acbc_310d_4ac4_ba1f_1f60fdcb95fb.slice/crio-4f91e353a53d91ef04e8d31d565c3667af442cba04c4036841189583de456ee9 WatchSource:0}: Error finding container 4f91e353a53d91ef04e8d31d565c3667af442cba04c4036841189583de456ee9: Status 404 returned error can't find the container with id 4f91e353a53d91ef04e8d31d565c3667af442cba04c4036841189583de456ee9 Oct 30 06:12:53.073657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:53.073620 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:53.073754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:53.073689 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:53.073754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:53.073720 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:53.124371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:53.124347 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:53.260206 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:53.260163 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" event={"ID":"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb","Type":"ContainerStarted","Data":"d1ccd3604e1835cb22be0eefc66f02dcc6e34371f5a24577b3acee0f011b186f"} Oct 30 06:12:53.260438 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:53.260263 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" event={"ID":"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb","Type":"ContainerStarted","Data":"4f91e353a53d91ef04e8d31d565c3667af442cba04c4036841189583de456ee9"} Oct 30 06:12:53.306468 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:53.306431 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:53.310369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:53.310307 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" podStartSLOduration=1.310285878 podStartE2EDuration="1.310285878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:53.309152003 +0000 UTC m=+1139.818171891" watchObservedRunningTime="2026-10-30 06:12:53.310285878 +0000 UTC m=+1139.819305775" Oct 30 06:12:53.727885 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:53.727851 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:12:53.804382 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:53.804353 6999 scope.go:122] "RemoveContainer" containerID="6929b8fb11437af442a6de8b9dd68e8df559187288e8f798ba305872ec1c0ed1" Oct 30 06:12:53.814770 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:53.814732 6999 scope.go:122] "RemoveContainer" containerID="3e7abad5261db035f23949ede9446f99c4a071fedb05487e7f06b764bfc79ce9" Oct 30 06:12:53.829132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:53.829095 6999 scope.go:122] "RemoveContainer" containerID="38fab71062104b8b32b17177e1bf9870e54683eb37a3a171749dba24d48cea3a" Oct 30 06:12:54.256239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:54.255283 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:54.729399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:54.729358 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:12:54.729797 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:54.729407 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" podUID="f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 06:12:55.727583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:55.727554 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:12:55.727801 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:55.727603 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" podUID="f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 06:12:56.260621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:56.260592 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:56.727078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:56.727043 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:12:56.727309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:56.727103 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" podUID="f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 06:12:57.727837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:57.727800 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:12:57.728289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:57.727878 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" podUID="f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 06:12:58.343661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:58.343628 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:58.504394 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:58.503550 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:58.727302 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:58.727258 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:12:58.727482 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:58.727315 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" podUID="f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 06:12:58.974939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:58.974911 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:59.536250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:59.536183 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:12:59.728017 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:59.727980 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 06:12:59.728188 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:12:59.728049 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" podUID="f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 06:13:00.024082 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:00.024048 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:00.181632 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:00.181597 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:00.608090 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:00.608061 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:00.750910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:00.750877 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:13:00.751899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:00.751873 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:13:00.753098 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:00.753068 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:13:00.756412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:00.756388 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:02.382652 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:02.382623 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:04.143281 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:13:04.143204 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:13:04.226131 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:04.226099 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:06.013879 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:06.013822 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:07.728807 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:07.728769 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:09.523825 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:09.523792 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:11.467071 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:11.467029 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:13.179945 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.179916 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph"] Oct 30 06:13:13.184312 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.184293 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:13.213756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.213730 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f789d7-serving-cert\"" Oct 30 06:13:13.218423 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.218397 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph"] Oct 30 06:13:13.309294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.309266 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-metrics-certs\") pod \"maas-api-e2e-api-b-f789d7-5465df4f74-bgrph\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:13.309445 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.309307 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-maas-api-tls\") pod \"maas-api-e2e-api-b-f789d7-5465df4f74-bgrph\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:13.309445 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.309416 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-openshift-service-ca\") pod \"maas-api-e2e-api-b-f789d7-5465df4f74-bgrph\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:13.309569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.309555 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tfw7\" (UniqueName: \"kubernetes.io/projected/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-kube-api-access-2tfw7\") pod \"maas-api-e2e-api-b-f789d7-5465df4f74-bgrph\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:13.314579 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.314551 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:13.411158 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.411131 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tfw7\" (UniqueName: \"kubernetes.io/projected/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-kube-api-access-2tfw7\") pod \"maas-api-e2e-api-b-f789d7-5465df4f74-bgrph\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:13.411355 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.411275 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-metrics-certs\") pod \"maas-api-e2e-api-b-f789d7-5465df4f74-bgrph\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:13.411355 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.411309 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-maas-api-tls\") pod \"maas-api-e2e-api-b-f789d7-5465df4f74-bgrph\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:13.411470 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.411359 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-openshift-service-ca\") pod \"maas-api-e2e-api-b-f789d7-5465df4f74-bgrph\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:13.412080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.412021 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-openshift-service-ca\") pod \"maas-api-e2e-api-b-f789d7-5465df4f74-bgrph\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:13.413896 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.413872 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-metrics-certs\") pod \"maas-api-e2e-api-b-f789d7-5465df4f74-bgrph\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:13.414199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.414180 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-maas-api-tls\") pod \"maas-api-e2e-api-b-f789d7-5465df4f74-bgrph\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:13.434625 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.434569 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tfw7\" (UniqueName: \"kubernetes.io/projected/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-kube-api-access-2tfw7\") pod \"maas-api-e2e-api-b-f789d7-5465df4f74-bgrph\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:13.456118 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.456094 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:13.497793 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.497760 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:13.608679 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.608642 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:13.824297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:13.824266 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph"] Oct 30 06:13:13.824821 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:13:13.824801 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f91b7e0_1276_4142_8b6a_eca9a5cdb1d6.slice/crio-4c4faa0c49409634dbb019538833a5f60395ef460918cca872a1a06606da2a3c WatchSource:0}: Error finding container 4c4faa0c49409634dbb019538833a5f60395ef460918cca872a1a06606da2a3c: Status 404 returned error can't find the container with id 4c4faa0c49409634dbb019538833a5f60395ef460918cca872a1a06606da2a3c Oct 30 06:13:14.456448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:14.456411 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" event={"ID":"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6","Type":"ContainerStarted","Data":"0e249a7958506e4147d91a58e984d7bcc1cf3367e339c2ce65cd37cf2ac63d23"} Oct 30 06:13:14.459048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:14.459022 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" event={"ID":"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6","Type":"ContainerStarted","Data":"4c4faa0c49409634dbb019538833a5f60395ef460918cca872a1a06606da2a3c"} Oct 30 06:13:14.459203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:14.459177 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:14.501380 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:14.501331 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" podStartSLOduration=1.034171779 podStartE2EDuration="1.501314771s" totalImagesPullingTime="467.142992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:14.498565805 +0000 UTC m=+1161.007585692" watchObservedRunningTime="2026-10-30 06:13:14.501314771 +0000 UTC m=+1161.010334659" Oct 30 06:13:15.228408 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:15.228378 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:17.115035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:17.115001 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:18.825409 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:18.825377 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:20.674093 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:20.674062 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:21.474265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:21.474203 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:13:22.459660 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:22.459616 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:23.595853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:23.595821 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:24.083288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:24.083255 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:24.258745 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:24.258711 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:24.510136 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:24.510006 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:26.004288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.004251 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq"] Oct 30 06:13:26.009030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.009003 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.040081 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.040051 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq"] Oct 30 06:13:26.178102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.178062 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.178335 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.178306 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.178428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.178348 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.178428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.178417 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.178522 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.178445 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.178578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.178566 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.178615 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.178589 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmqt\" (UniqueName: \"kubernetes.io/projected/4fb98d9c-92c6-4aae-b631-f8319a76ee40-kube-api-access-6hmqt\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.178658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.178620 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.178772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.178753 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.202913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.202876 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:26.280053 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.279954 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.280053 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.280009 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.280353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.280155 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.280353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.280192 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.280353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.280236 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.280353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.280341 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.280570 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.280493 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.280570 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.280527 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmqt\" (UniqueName: \"kubernetes.io/projected/4fb98d9c-92c6-4aae-b631-f8319a76ee40-kube-api-access-6hmqt\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.280676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.280575 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.281298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.280792 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-credential-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.281298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.280812 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-workload-socket\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.281298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.280912 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istiod-ca-cert\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.281298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.281124 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-data\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.281538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.281328 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-workload-certs\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.283420 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.283388 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-envoy\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.283786 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.283766 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-podinfo\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.303658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.303637 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-token\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.303767 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.303707 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmqt\" (UniqueName: \"kubernetes.io/projected/4fb98d9c-92c6-4aae-b631-f8319a76ee40-kube-api-access-6hmqt\") pod \"e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.328241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.328207 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:26.471197 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.471155 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:26.621974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.621940 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:26.654858 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:13:26.654830 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb98d9c_92c6_4aae_b631_f8319a76ee40.slice/crio-555542efd9b058aef1027775b3932b47455a1aa37ec7d4c20caa98e5fdc1f186 WatchSource:0}: Error finding container 555542efd9b058aef1027775b3932b47455a1aa37ec7d4c20caa98e5fdc1f186: Status 404 returned error can't find the container with id 555542efd9b058aef1027775b3932b47455a1aa37ec7d4c20caa98e5fdc1f186 Oct 30 06:13:26.655267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.655246 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq"] Oct 30 06:13:26.657488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.657454 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:26.657590 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.657535 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:26.657590 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:26.657582 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:27.556292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:27.556253 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" event={"ID":"4fb98d9c-92c6-4aae-b631-f8319a76ee40","Type":"ContainerStarted","Data":"2ca9095be0fa03a7e0dfe51d8568f0de1ef5a69e08115dac0df393a593948b83"} Oct 30 06:13:27.557012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:27.556909 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" event={"ID":"4fb98d9c-92c6-4aae-b631-f8319a76ee40","Type":"ContainerStarted","Data":"555542efd9b058aef1027775b3932b47455a1aa37ec7d4c20caa98e5fdc1f186"} Oct 30 06:13:27.617717 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:27.617665 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" podStartSLOduration=2.617652304 podStartE2EDuration="2.617652304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:27.615711105 +0000 UTC m=+1174.124730994" watchObservedRunningTime="2026-10-30 06:13:27.617652304 +0000 UTC m=+1174.126672191" Oct 30 06:13:28.172017 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:28.171989 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:28.334241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:28.333562 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:28.367238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:28.362519 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 06:13:28.367238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:28.362582 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" podUID="4fb98d9c-92c6-4aae-b631-f8319a76ee40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 06:13:28.463905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:28.463805 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:29.329032 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:29.329002 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 06:13:29.329431 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:29.329063 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" podUID="4fb98d9c-92c6-4aae-b631-f8319a76ee40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 06:13:29.882226 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:29.882179 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:30.039752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.039697 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:30.127318 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.127285 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw"] Oct 30 06:13:30.148962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.148886 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" Oct 30 06:13:30.173927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.173893 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcsmb\" (UniqueName: \"kubernetes.io/projected/6815f024-a76f-45e1-b16d-c1d3ed0921b3-kube-api-access-bcsmb\") pod \"payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw\" (UID: \"6815f024-a76f-45e1-b16d-c1d3ed0921b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" Oct 30 06:13:30.174455 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.174427 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6815f024-a76f-45e1-b16d-c1d3ed0921b3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw\" (UID: \"6815f024-a76f-45e1-b16d-c1d3ed0921b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" Oct 30 06:13:30.184130 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.181517 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e6df62-dockercfg-bfdw8\"" Oct 30 06:13:30.244944 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.244912 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e6df62\"" Oct 30 06:13:30.277634 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.277602 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6815f024-a76f-45e1-b16d-c1d3ed0921b3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw\" (UID: \"6815f024-a76f-45e1-b16d-c1d3ed0921b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" Oct 30 06:13:30.278061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.278041 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcsmb\" (UniqueName: \"kubernetes.io/projected/6815f024-a76f-45e1-b16d-c1d3ed0921b3-kube-api-access-bcsmb\") pod \"payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw\" (UID: \"6815f024-a76f-45e1-b16d-c1d3ed0921b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" Oct 30 06:13:30.279759 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.279734 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6815f024-a76f-45e1-b16d-c1d3ed0921b3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw\" (UID: \"6815f024-a76f-45e1-b16d-c1d3ed0921b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" Oct 30 06:13:30.286342 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.286261 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw"] Oct 30 06:13:30.315964 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.315912 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcsmb\" (UniqueName: \"kubernetes.io/projected/6815f024-a76f-45e1-b16d-c1d3ed0921b3-kube-api-access-bcsmb\") pod \"payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw\" (UID: \"6815f024-a76f-45e1-b16d-c1d3ed0921b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" Oct 30 06:13:30.328685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.328655 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 06:13:30.328851 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.328712 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" podUID="4fb98d9c-92c6-4aae-b631-f8319a76ee40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 06:13:30.357185 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.357036 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:30.580805 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.580722 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" Oct 30 06:13:30.924117 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:30.924088 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw"] Oct 30 06:13:30.925416 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:13:30.925392 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6815f024_a76f_45e1_b16d_c1d3ed0921b3.slice/crio-843cd0c335eda0a643dd35e447b4bff1bb8d178deade07c608611a390c03cc6c WatchSource:0}: Error finding container 843cd0c335eda0a643dd35e447b4bff1bb8d178deade07c608611a390c03cc6c: Status 404 returned error can't find the container with id 843cd0c335eda0a643dd35e447b4bff1bb8d178deade07c608611a390c03cc6c Oct 30 06:13:31.328881 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:31.328798 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 06:13:31.328881 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:31.328845 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" podUID="4fb98d9c-92c6-4aae-b631-f8319a76ee40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 06:13:31.588839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:31.588740 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" event={"ID":"6815f024-a76f-45e1-b16d-c1d3ed0921b3","Type":"ContainerStarted","Data":"50dd2b211b0a5b3afc88fdb0f6c2e414096c01a17e1cfb9be09d39cb4e560c28"} Oct 30 06:13:31.588839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:31.588803 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" event={"ID":"6815f024-a76f-45e1-b16d-c1d3ed0921b3","Type":"ContainerStarted","Data":"843cd0c335eda0a643dd35e447b4bff1bb8d178deade07c608611a390c03cc6c"} Oct 30 06:13:31.589376 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:31.588844 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" Oct 30 06:13:31.632554 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:31.632500 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" podStartSLOduration=1.6324869770000001 podStartE2EDuration="1.632486977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:31.629913005 +0000 UTC m=+1178.138932892" watchObservedRunningTime="2026-10-30 06:13:31.632486977 +0000 UTC m=+1178.141506863" Oct 30 06:13:32.255239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:32.254241 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:32.328723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:32.328686 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 06:13:32.328892 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:32.328737 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" podUID="4fb98d9c-92c6-4aae-b631-f8319a76ee40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 06:13:33.328739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:33.328704 6999 patch_prober.go:29] interesting pod/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 06:13:33.329123 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:33.328767 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" podUID="4fb98d9c-92c6-4aae-b631-f8319a76ee40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 06:13:33.809390 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:33.809357 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:34.165486 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:13:34.165447 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:13:34.314727 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:34.314683 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:34.336403 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:34.336365 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:34.337478 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:34.337455 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:34.338428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:34.338408 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:13:35.214558 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:35.214516 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:36.473238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:36.473115 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:38.543413 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:38.543377 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:40.336648 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:40.336619 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:41.728375 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:41.728345 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:42.137437 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:42.137412 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:42.598343 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:42.598274 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" Oct 30 06:13:44.065914 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:44.065879 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:45.608569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.608542 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj"] Oct 30 06:13:45.612748 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.612730 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:45.627281 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.627261 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e6df62-serving-cert\"" Oct 30 06:13:45.627803 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.627782 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj"] Oct 30 06:13:45.706487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.706457 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-metrics-certs\") pod \"maas-api-e2e-api-a-e6df62-8559497db9-7vtvj\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:45.706692 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.706668 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxktz\" (UniqueName: \"kubernetes.io/projected/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-kube-api-access-lxktz\") pod \"maas-api-e2e-api-a-e6df62-8559497db9-7vtvj\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:45.706876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.706854 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-maas-api-tls\") pod \"maas-api-e2e-api-a-e6df62-8559497db9-7vtvj\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:45.707106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.707085 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-openshift-service-ca\") pod \"maas-api-e2e-api-a-e6df62-8559497db9-7vtvj\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:45.731429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.731399 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:45.807734 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.807706 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxktz\" (UniqueName: \"kubernetes.io/projected/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-kube-api-access-lxktz\") pod \"maas-api-e2e-api-a-e6df62-8559497db9-7vtvj\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:45.807862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.807779 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-maas-api-tls\") pod \"maas-api-e2e-api-a-e6df62-8559497db9-7vtvj\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:45.807862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.807854 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-openshift-service-ca\") pod \"maas-api-e2e-api-a-e6df62-8559497db9-7vtvj\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:45.807954 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.807929 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-metrics-certs\") pod \"maas-api-e2e-api-a-e6df62-8559497db9-7vtvj\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:45.808504 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.808482 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-openshift-service-ca\") pod \"maas-api-e2e-api-a-e6df62-8559497db9-7vtvj\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:45.810304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.810283 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-metrics-certs\") pod \"maas-api-e2e-api-a-e6df62-8559497db9-7vtvj\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:45.810418 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.810322 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-maas-api-tls\") pod \"maas-api-e2e-api-a-e6df62-8559497db9-7vtvj\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:45.831021 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.831000 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxktz\" (UniqueName: \"kubernetes.io/projected/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-kube-api-access-lxktz\") pod \"maas-api-e2e-api-a-e6df62-8559497db9-7vtvj\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:45.873566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.873474 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:45.926318 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:45.926290 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:46.033097 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:46.033037 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:46.252884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:46.252850 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj"] Oct 30 06:13:46.254087 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:13:46.254059 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bcbb1eb_21c8_4b95_a209_8d94d54f719d.slice/crio-14a211b5d32f654103739a14d4f03fe664367119e25371cfc99d4fed98cb6cf2 WatchSource:0}: Error finding container 14a211b5d32f654103739a14d4f03fe664367119e25371cfc99d4fed98cb6cf2: Status 404 returned error can't find the container with id 14a211b5d32f654103739a14d4f03fe664367119e25371cfc99d4fed98cb6cf2 Oct 30 06:13:46.526757 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:46.526675 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:46.706004 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:46.702099 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" event={"ID":"1bcbb1eb-21c8-4b95-a209-8d94d54f719d","Type":"ContainerStarted","Data":"c9cd77fd96f1dc7794f933349cec8d044751e1070a2f449526e678f9dd236fc9"} Oct 30 06:13:46.706004 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:46.702163 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" event={"ID":"1bcbb1eb-21c8-4b95-a209-8d94d54f719d","Type":"ContainerStarted","Data":"14a211b5d32f654103739a14d4f03fe664367119e25371cfc99d4fed98cb6cf2"} Oct 30 06:13:46.706004 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:46.702424 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:46.748321 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:46.748276 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" podStartSLOduration=1.400289729 podStartE2EDuration="1.74825775s" totalImagesPullingTime="347.968021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:46.744569359 +0000 UTC m=+1193.253589246" watchObservedRunningTime="2026-10-30 06:13:46.74825775 +0000 UTC m=+1193.257277631" Oct 30 06:13:46.749800 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:46.749771 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:47.143159 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:47.143130 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7318-884907"] Oct 30 06:13:47.147933 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:47.147913 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7318-884907" Oct 30 06:13:47.161411 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:47.161373 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7318-884907"] Oct 30 06:13:47.163275 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:47.163254 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p6zmk\"" Oct 30 06:13:47.240762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:47.240723 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbbxf\" (UniqueName: \"kubernetes.io/projected/a6742db8-11ab-42ec-abd9-4abd105f8435-kube-api-access-jbbxf\") pod \"test-curl-7318-884907\" (UID: \"a6742db8-11ab-42ec-abd9-4abd105f8435\") " pod="opendatahub/test-curl-7318-884907" Oct 30 06:13:47.341838 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:47.341804 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbbxf\" (UniqueName: \"kubernetes.io/projected/a6742db8-11ab-42ec-abd9-4abd105f8435-kube-api-access-jbbxf\") pod \"test-curl-7318-884907\" (UID: \"a6742db8-11ab-42ec-abd9-4abd105f8435\") " pod="opendatahub/test-curl-7318-884907" Oct 30 06:13:47.366013 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:47.365982 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbbxf\" (UniqueName: \"kubernetes.io/projected/a6742db8-11ab-42ec-abd9-4abd105f8435-kube-api-access-jbbxf\") pod \"test-curl-7318-884907\" (UID: \"a6742db8-11ab-42ec-abd9-4abd105f8435\") " pod="opendatahub/test-curl-7318-884907" Oct 30 06:13:47.464482 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:47.464405 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7318-884907" Oct 30 06:13:47.698908 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:47.698872 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:47.774277 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:47.773201 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7318-884907"] Oct 30 06:13:47.775038 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:13:47.775009 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6742db8_11ab_42ec_abd9_4abd105f8435.slice/crio-cb261ac4bb7ff17e694eec6cab06135f197fa80fd530aa011f8374aebdf57979 WatchSource:0}: Error finding container cb261ac4bb7ff17e694eec6cab06135f197fa80fd530aa011f8374aebdf57979: Status 404 returned error can't find the container with id cb261ac4bb7ff17e694eec6cab06135f197fa80fd530aa011f8374aebdf57979 Oct 30 06:13:48.717380 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:48.717284 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7318-884907" event={"ID":"a6742db8-11ab-42ec-abd9-4abd105f8435","Type":"ContainerStarted","Data":"5b781cf369179a106852e90fac1b43646601bc8ffaaba93b16b967878d6660a2"} Oct 30 06:13:48.717380 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:48.717335 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7318-884907" event={"ID":"a6742db8-11ab-42ec-abd9-4abd105f8435","Type":"ContainerStarted","Data":"cb261ac4bb7ff17e694eec6cab06135f197fa80fd530aa011f8374aebdf57979"} Oct 30 06:13:48.755182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:48.755131 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7318-884907" podStartSLOduration=1.157276746 podStartE2EDuration="1.7551194s" totalImagesPullingTime="597.842654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:48.754595164 +0000 UTC m=+1195.263615051" watchObservedRunningTime="2026-10-30 06:13:48.7551194 +0000 UTC m=+1195.264139286" Oct 30 06:13:49.549019 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:49.548971 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:49.664199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:49.664167 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7318-884907"] Oct 30 06:13:49.670716 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:49.670687 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7318-884907"] Oct 30 06:13:50.730507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:50.730472 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7318-884907" podUID="a6742db8-11ab-42ec-abd9-4abd105f8435" containerName="test-curl-7318-884907" containerID="cri-o://5b781cf369179a106852e90fac1b43646601bc8ffaaba93b16b967878d6660a2" gracePeriod=2 Oct 30 06:13:50.744027 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:13:50.743998 6999 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7318-884907\" is forbidden: User \"system:node:ip-10-0-131-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-226.ec2.internal' and this object" podUID="a6742db8-11ab-42ec-abd9-4abd105f8435" pod="opendatahub/test-curl-7318-884907" Oct 30 06:13:51.443562 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:51.443530 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:52.715408 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:52.715371 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:13:52.951120 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:52.951092 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7318-884907" Oct 30 06:13:53.026412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:53.026338 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6742db8-11ab-42ec-abd9-4abd105f8435-kube-api-access-jbbxf\" (UniqueName: \"kubernetes.io/projected/a6742db8-11ab-42ec-abd9-4abd105f8435-kube-api-access-jbbxf\") pod \"a6742db8-11ab-42ec-abd9-4abd105f8435\" (UID: \"a6742db8-11ab-42ec-abd9-4abd105f8435\") " Oct 30 06:13:53.028579 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:53.028555 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6742db8-11ab-42ec-abd9-4abd105f8435-kube-api-access-jbbxf" pod "a6742db8-11ab-42ec-abd9-4abd105f8435" (UID: "a6742db8-11ab-42ec-abd9-4abd105f8435"). InnerVolumeSpecName "kube-api-access-jbbxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:53.128381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:53.128359 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbbxf\" (UniqueName: \"kubernetes.io/projected/a6742db8-11ab-42ec-abd9-4abd105f8435-kube-api-access-jbbxf\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:13:53.241906 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:53.241876 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:53.609301 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:53.609272 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6742db8-11ab-42ec-abd9-4abd105f8435" path="/var/lib/kubelet/pods/a6742db8-11ab-42ec-abd9-4abd105f8435/volumes" Oct 30 06:13:53.750464 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:53.750425 6999 generic.go:356] "Generic (PLEG): container finished" podID="a6742db8-11ab-42ec-abd9-4abd105f8435" containerID="5b781cf369179a106852e90fac1b43646601bc8ffaaba93b16b967878d6660a2" exitCode=137 Oct 30 06:13:53.750923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:53.750467 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7318-884907" Oct 30 06:13:53.750923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:53.750502 6999 scope.go:122] "RemoveContainer" containerID="5b781cf369179a106852e90fac1b43646601bc8ffaaba93b16b967878d6660a2" Oct 30 06:13:53.761157 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:53.761143 6999 scope.go:122] "RemoveContainer" containerID="5b781cf369179a106852e90fac1b43646601bc8ffaaba93b16b967878d6660a2" Oct 30 06:13:53.761418 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:13:53.761398 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b781cf369179a106852e90fac1b43646601bc8ffaaba93b16b967878d6660a2\": container with ID starting with 5b781cf369179a106852e90fac1b43646601bc8ffaaba93b16b967878d6660a2 not found: ID does not exist" containerID="5b781cf369179a106852e90fac1b43646601bc8ffaaba93b16b967878d6660a2" Oct 30 06:13:53.761511 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:53.761431 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b781cf369179a106852e90fac1b43646601bc8ffaaba93b16b967878d6660a2"} err="failed to get container status \"5b781cf369179a106852e90fac1b43646601bc8ffaaba93b16b967878d6660a2\": rpc error: code = NotFound desc = could not find container \"5b781cf369179a106852e90fac1b43646601bc8ffaaba93b16b967878d6660a2\": container with ID starting with 5b781cf369179a106852e90fac1b43646601bc8ffaaba93b16b967878d6660a2 not found: ID does not exist" Oct 30 06:13:55.083666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:55.083571 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:56.820159 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:56.820129 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:58.644736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:58.644703 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:59.145958 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:59.145917 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:13:59.791921 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:13:59.791887 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:00.546940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:00.546898 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:01.041237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:01.041102 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:01.383263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:01.379262 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:01.585239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:01.584823 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:02.046996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:02.046962 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt"] Oct 30 06:14:02.070236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:02.068770 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc9fd2f-fjwmt"] Oct 30 06:14:02.857826 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:02.857800 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:03.610602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:03.610568 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77704d62-3fd8-428d-b998-cceb2d742116" path="/var/lib/kubelet/pods/77704d62-3fd8-428d-b998-cceb2d742116/volumes" Oct 30 06:14:04.143460 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:14:04.143421 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:14:04.644945 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:04.644917 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:06.552079 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:06.552028 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:08.442089 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:08.442054 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:08.665561 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:08.665532 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:09.183733 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.183701 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d"] Oct 30 06:14:09.184323 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.184310 6999 cpu_manager.go:414] "containerMap: removing container" podUID="a6742db8-11ab-42ec-abd9-4abd105f8435" containerName="test-curl-7318-884907" Oct 30 06:14:09.184380 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.184325 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6742db8-11ab-42ec-abd9-4abd105f8435" containerName="test-curl-7318-884907" Oct 30 06:14:09.184434 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.184425 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6742db8-11ab-42ec-abd9-4abd105f8435" containerName="test-curl-7318-884907" Oct 30 06:14:09.189530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.189511 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.220862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.220836 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d"] Oct 30 06:14:09.268708 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.268681 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-envoy\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.268833 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.268737 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-workload-certs\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.268918 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.268849 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-credential-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.268918 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.268905 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-workload-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.269023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.268937 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9cgw\" (UniqueName: \"kubernetes.io/projected/c4989e46-f4b8-4113-aea1-66994ea9c6c6-kube-api-access-q9cgw\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.269112 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.269097 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-token\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.269185 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.269133 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-data\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.269185 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.269158 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-podinfo\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.269288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.269241 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istiod-ca-cert\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.295540 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.295519 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e6df62-openshift-default-dockercfg-9d68h\"" Oct 30 06:14:09.370305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.370276 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istiod-ca-cert\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.370428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.370397 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-envoy\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.370497 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.370459 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-workload-certs\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.370553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.370498 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-credential-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.370610 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.370550 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-workload-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.370610 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.370579 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9cgw\" (UniqueName: \"kubernetes.io/projected/c4989e46-f4b8-4113-aea1-66994ea9c6c6-kube-api-access-q9cgw\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.370743 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.370726 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-token\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.370838 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.370815 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-workload-certs\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.370987 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.370820 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-data\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.371083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.371007 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-podinfo\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.371083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.370869 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-workload-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.371083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.371068 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-data\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.371083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.370886 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-credential-socket\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.371849 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.371260 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istiod-ca-cert\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.372663 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.372637 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-envoy\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.373303 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.373285 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-podinfo\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.381926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.381907 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:09.393788 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.393770 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-token\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.393895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.393821 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9cgw\" (UniqueName: \"kubernetes.io/projected/c4989e46-f4b8-4113-aea1-66994ea9c6c6-kube-api-access-q9cgw\") pod \"e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.504533 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.504478 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:09.834335 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.834303 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d"] Oct 30 06:14:09.837529 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:14:09.837501 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4989e46_f4b8_4113_aea1_66994ea9c6c6.slice/crio-cdadbac51865e52550f671b66fe6d9c102b8a48d6e97f7bb04d2f8c4ad45a23a WatchSource:0}: Error finding container cdadbac51865e52550f671b66fe6d9c102b8a48d6e97f7bb04d2f8c4ad45a23a: Status 404 returned error can't find the container with id cdadbac51865e52550f671b66fe6d9c102b8a48d6e97f7bb04d2f8c4ad45a23a Oct 30 06:14:09.840423 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.840382 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:09.840524 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.840470 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:09.840524 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.840515 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:09.889931 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.889901 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:09.911495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:09.911387 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" event={"ID":"c4989e46-f4b8-4113-aea1-66994ea9c6c6","Type":"ContainerStarted","Data":"cdadbac51865e52550f671b66fe6d9c102b8a48d6e97f7bb04d2f8c4ad45a23a"} Oct 30 06:14:10.344059 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:10.344020 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:10.511379 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:10.511346 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:10.921759 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:10.921724 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" event={"ID":"c4989e46-f4b8-4113-aea1-66994ea9c6c6","Type":"ContainerStarted","Data":"2c6553135b536ef6c7622ca4109a42165dccc521cfe3afbb0507826cd9bf17b9"} Oct 30 06:14:10.996177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:10.996121 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" podStartSLOduration=1.99610791 podStartE2EDuration="1.99610791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:10.981732932 +0000 UTC m=+1217.490752833" watchObservedRunningTime="2026-10-30 06:14:10.99610791 +0000 UTC m=+1217.505127842" Oct 30 06:14:11.504880 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:11.504849 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:11.506046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:11.506024 6999 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 06:14:11.506174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:11.506074 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" podUID="c4989e46-f4b8-4113-aea1-66994ea9c6c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 06:14:12.267256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:12.265652 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:12.505292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:12.505253 6999 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 06:14:12.505436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:12.505332 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" podUID="c4989e46-f4b8-4113-aea1-66994ea9c6c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 06:14:13.505261 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:13.505206 6999 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 06:14:13.505708 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:13.505286 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" podUID="c4989e46-f4b8-4113-aea1-66994ea9c6c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 06:14:14.379530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:14.379494 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:14.505174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:14.505137 6999 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 06:14:14.505410 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:14.505280 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" podUID="c4989e46-f4b8-4113-aea1-66994ea9c6c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 06:14:14.559238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:14.555636 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:14.730593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:14.730522 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:15.504934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:15.504891 6999 patch_prober.go:29] interesting pod/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 06:14:15.505116 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:15.504952 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" podUID="c4989e46-f4b8-4113-aea1-66994ea9c6c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 06:14:16.307098 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:16.307059 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:16.460141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:16.460106 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:16.511504 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:16.511457 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:16.514325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:16.514280 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:16.517497 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:16.517468 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:14:16.727285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:16.727259 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:18.223876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.223817 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf"] Oct 30 06:14:18.224610 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.224575 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" podUID="78e8576b-5ebe-47fa-aeb5-d63701c9018d" containerName="maas-api" containerID="cri-o://05a5dce0e31c24d9df55d4f70683bfd9d80a3686ff5884242ecf68dd2957a876" gracePeriod=30 Oct 30 06:14:18.324547 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.324515 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:18.564850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.564412 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:14:18.651685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.651655 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78e8576b-5ebe-47fa-aeb5-d63701c9018d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e8576b-5ebe-47fa-aeb5-d63701c9018d-metrics-certs\") pod \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " Oct 30 06:14:18.651870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.651718 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e8576b-5ebe-47fa-aeb5-d63701c9018d-kube-api-access-cfbbg\" (UniqueName: \"kubernetes.io/projected/78e8576b-5ebe-47fa-aeb5-d63701c9018d-kube-api-access-cfbbg\") pod \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " Oct 30 06:14:18.651870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.651804 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e8576b-5ebe-47fa-aeb5-d63701c9018d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e8576b-5ebe-47fa-aeb5-d63701c9018d-openshift-service-ca\") pod \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " Oct 30 06:14:18.652018 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.651951 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78e8576b-5ebe-47fa-aeb5-d63701c9018d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78e8576b-5ebe-47fa-aeb5-d63701c9018d-maas-api-tls\") pod \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\" (UID: \"78e8576b-5ebe-47fa-aeb5-d63701c9018d\") " Oct 30 06:14:18.652186 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.652157 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e8576b-5ebe-47fa-aeb5-d63701c9018d-openshift-service-ca" pod "78e8576b-5ebe-47fa-aeb5-d63701c9018d" (UID: "78e8576b-5ebe-47fa-aeb5-d63701c9018d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:18.652960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.652876 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e8576b-5ebe-47fa-aeb5-d63701c9018d-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:14:18.654495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.654474 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e8576b-5ebe-47fa-aeb5-d63701c9018d-metrics-certs" pod "78e8576b-5ebe-47fa-aeb5-d63701c9018d" (UID: "78e8576b-5ebe-47fa-aeb5-d63701c9018d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:18.654718 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.654688 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78e8576b-5ebe-47fa-aeb5-d63701c9018d-maas-api-tls" pod "78e8576b-5ebe-47fa-aeb5-d63701c9018d" (UID: "78e8576b-5ebe-47fa-aeb5-d63701c9018d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:18.655137 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.655118 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e8576b-5ebe-47fa-aeb5-d63701c9018d-kube-api-access-cfbbg" pod "78e8576b-5ebe-47fa-aeb5-d63701c9018d" (UID: "78e8576b-5ebe-47fa-aeb5-d63701c9018d"). InnerVolumeSpecName "kube-api-access-cfbbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:18.753932 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.753900 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78e8576b-5ebe-47fa-aeb5-d63701c9018d-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:14:18.753932 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.753928 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78e8576b-5ebe-47fa-aeb5-d63701c9018d-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:14:18.753932 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.753939 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfbbg\" (UniqueName: \"kubernetes.io/projected/78e8576b-5ebe-47fa-aeb5-d63701c9018d-kube-api-access-cfbbg\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:14:18.902086 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:18.902044 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:19.005239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:19.004299 6999 generic.go:356] "Generic (PLEG): container finished" podID="78e8576b-5ebe-47fa-aeb5-d63701c9018d" containerID="05a5dce0e31c24d9df55d4f70683bfd9d80a3686ff5884242ecf68dd2957a876" exitCode=0 Oct 30 06:14:19.005239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:19.004349 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" event={"ID":"78e8576b-5ebe-47fa-aeb5-d63701c9018d","Type":"ContainerDied","Data":"05a5dce0e31c24d9df55d4f70683bfd9d80a3686ff5884242ecf68dd2957a876"} Oct 30 06:14:19.005239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:19.004379 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" Oct 30 06:14:19.005239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:19.004411 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf" event={"ID":"78e8576b-5ebe-47fa-aeb5-d63701c9018d","Type":"ContainerDied","Data":"92410a6788dd695e8be2eaf5b9dcec73747f49797b624af9d8308f48c9275bfe"} Oct 30 06:14:19.005239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:19.004440 6999 scope.go:122] "RemoveContainer" containerID="05a5dce0e31c24d9df55d4f70683bfd9d80a3686ff5884242ecf68dd2957a876" Oct 30 06:14:19.018086 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:19.018061 6999 scope.go:122] "RemoveContainer" containerID="05a5dce0e31c24d9df55d4f70683bfd9d80a3686ff5884242ecf68dd2957a876" Oct 30 06:14:19.018503 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:14:19.018462 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a5dce0e31c24d9df55d4f70683bfd9d80a3686ff5884242ecf68dd2957a876\": container with ID starting with 05a5dce0e31c24d9df55d4f70683bfd9d80a3686ff5884242ecf68dd2957a876 not found: ID does not exist" containerID="05a5dce0e31c24d9df55d4f70683bfd9d80a3686ff5884242ecf68dd2957a876" Oct 30 06:14:19.018619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:19.018517 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a5dce0e31c24d9df55d4f70683bfd9d80a3686ff5884242ecf68dd2957a876"} err="failed to get container status \"05a5dce0e31c24d9df55d4f70683bfd9d80a3686ff5884242ecf68dd2957a876\": rpc error: code = NotFound desc = could not find container \"05a5dce0e31c24d9df55d4f70683bfd9d80a3686ff5884242ecf68dd2957a876\": container with ID starting with 05a5dce0e31c24d9df55d4f70683bfd9d80a3686ff5884242ecf68dd2957a876 not found: ID does not exist" Oct 30 06:14:19.065320 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:19.065293 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf"] Oct 30 06:14:19.069501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:19.069465 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:19.076009 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:19.075984 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-1ec766-5cc6d67988-h4hpf"] Oct 30 06:14:19.233789 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:19.233683 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:19.602972 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:19.602887 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:19.610014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:19.609986 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e8576b-5ebe-47fa-aeb5-d63701c9018d" path="/var/lib/kubelet/pods/78e8576b-5ebe-47fa-aeb5-d63701c9018d/volumes" Oct 30 06:14:19.760238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:19.757059 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:20.035195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:20.035155 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh"] Oct 30 06:14:20.050547 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:20.050514 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ca1629b-dn5dh"] Oct 30 06:14:21.083617 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:21.083581 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:21.614547 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:21.614507 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0966f3b6-5625-4967-a6b7-64289ea9f9b8" path="/var/lib/kubelet/pods/0966f3b6-5625-4967-a6b7-64289ea9f9b8/volumes" Oct 30 06:14:23.037068 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:23.037040 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:23.836267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:23.836236 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:23.992675 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:23.992646 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:25.096176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.096124 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:25.568065 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.568029 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:25.748018 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.747986 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw"] Oct 30 06:14:25.748664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.748648 6999 cpu_manager.go:414] "containerMap: removing container" podUID="78e8576b-5ebe-47fa-aeb5-d63701c9018d" containerName="maas-api" Oct 30 06:14:25.748725 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.748666 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e8576b-5ebe-47fa-aeb5-d63701c9018d" containerName="maas-api" Oct 30 06:14:25.748834 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.748816 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e8576b-5ebe-47fa-aeb5-d63701c9018d" containerName="maas-api" Oct 30 06:14:25.754016 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.753998 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" Oct 30 06:14:25.785579 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.785553 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c15d3a\"" Oct 30 06:14:25.797817 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.797798 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c15d3a-dockercfg-xhwp8\"" Oct 30 06:14:25.812178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.812143 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw"] Oct 30 06:14:25.856414 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.856330 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwb2z\" (UniqueName: \"kubernetes.io/projected/ff79aff8-16c9-4edf-8b16-81ef85272ae5-kube-api-access-mwb2z\") pod \"payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw\" (UID: \"ff79aff8-16c9-4edf-8b16-81ef85272ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" Oct 30 06:14:25.856551 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.856524 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff79aff8-16c9-4edf-8b16-81ef85272ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw\" (UID: \"ff79aff8-16c9-4edf-8b16-81ef85272ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" Oct 30 06:14:25.876939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.876904 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:25.957964 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.957926 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff79aff8-16c9-4edf-8b16-81ef85272ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw\" (UID: \"ff79aff8-16c9-4edf-8b16-81ef85272ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" Oct 30 06:14:25.958200 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.958182 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwb2z\" (UniqueName: \"kubernetes.io/projected/ff79aff8-16c9-4edf-8b16-81ef85272ae5-kube-api-access-mwb2z\") pod \"payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw\" (UID: \"ff79aff8-16c9-4edf-8b16-81ef85272ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" Oct 30 06:14:25.959615 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.959581 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff79aff8-16c9-4edf-8b16-81ef85272ae5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw\" (UID: \"ff79aff8-16c9-4edf-8b16-81ef85272ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" Oct 30 06:14:25.989939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:25.989902 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwb2z\" (UniqueName: \"kubernetes.io/projected/ff79aff8-16c9-4edf-8b16-81ef85272ae5-kube-api-access-mwb2z\") pod \"payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw\" (UID: \"ff79aff8-16c9-4edf-8b16-81ef85272ae5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" Oct 30 06:14:26.068313 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:26.067947 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" Oct 30 06:14:26.392182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:26.392151 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw"] Oct 30 06:14:26.393627 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:14:26.393603 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff79aff8_16c9_4edf_8b16_81ef85272ae5.slice/crio-12e27c026a5ba885f073ad35f19684681925f8e2a65f9cf712bd7e231f1c5bf3 WatchSource:0}: Error finding container 12e27c026a5ba885f073ad35f19684681925f8e2a65f9cf712bd7e231f1c5bf3: Status 404 returned error can't find the container with id 12e27c026a5ba885f073ad35f19684681925f8e2a65f9cf712bd7e231f1c5bf3 Oct 30 06:14:27.069177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:27.069144 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" event={"ID":"ff79aff8-16c9-4edf-8b16-81ef85272ae5","Type":"ContainerStarted","Data":"31cf9288793ca3b5edd4ab11565401ac47ee8b3bda495dff62d764b4cd5b07f0"} Oct 30 06:14:27.069406 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:27.069201 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" event={"ID":"ff79aff8-16c9-4edf-8b16-81ef85272ae5","Type":"ContainerStarted","Data":"12e27c026a5ba885f073ad35f19684681925f8e2a65f9cf712bd7e231f1c5bf3"} Oct 30 06:14:27.069406 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:27.069256 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" Oct 30 06:14:27.110674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:27.110624 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" podStartSLOduration=2.110610863 podStartE2EDuration="2.110610863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:27.109827437 +0000 UTC m=+1233.618847323" watchObservedRunningTime="2026-10-30 06:14:27.110610863 +0000 UTC m=+1233.619630758" Oct 30 06:14:27.220044 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:27.220015 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:27.506973 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:27.506940 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:27.657841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:27.657816 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:27.904327 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:27.904281 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:28.057936 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:28.057895 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:29.120401 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:29.120370 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:30.880051 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:30.880018 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs"] Oct 30 06:14:30.884774 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:30.884748 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:30.900476 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:30.900420 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-fe108e-serving-cert\"" Oct 30 06:14:30.929972 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:30.929938 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs"] Oct 30 06:14:31.013092 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.013048 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:31.038099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.038064 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186bbd33-30f6-4334-b87a-fa151cc6cd61-maas-api-tls\") pod \"maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:31.038269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.038117 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186bbd33-30f6-4334-b87a-fa151cc6cd61-metrics-certs\") pod \"maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:31.038269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.038184 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzvgx\" (UniqueName: \"kubernetes.io/projected/186bbd33-30f6-4334-b87a-fa151cc6cd61-kube-api-access-bzvgx\") pod \"maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:31.038452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.038432 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186bbd33-30f6-4334-b87a-fa151cc6cd61-openshift-service-ca\") pod \"maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:31.140385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.140264 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186bbd33-30f6-4334-b87a-fa151cc6cd61-openshift-service-ca\") pod \"maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:31.140385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.140348 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186bbd33-30f6-4334-b87a-fa151cc6cd61-maas-api-tls\") pod \"maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:31.140631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.140390 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186bbd33-30f6-4334-b87a-fa151cc6cd61-metrics-certs\") pod \"maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:31.140631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.140427 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzvgx\" (UniqueName: \"kubernetes.io/projected/186bbd33-30f6-4334-b87a-fa151cc6cd61-kube-api-access-bzvgx\") pod \"maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:31.141530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.141500 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186bbd33-30f6-4334-b87a-fa151cc6cd61-openshift-service-ca\") pod \"maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:31.143391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.143367 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186bbd33-30f6-4334-b87a-fa151cc6cd61-metrics-certs\") pod \"maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:31.143739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.143700 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186bbd33-30f6-4334-b87a-fa151cc6cd61-maas-api-tls\") pod \"maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:31.164266 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.164231 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:31.164542 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.164520 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzvgx\" (UniqueName: \"kubernetes.io/projected/186bbd33-30f6-4334-b87a-fa151cc6cd61-kube-api-access-bzvgx\") pod \"maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:31.200465 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.200428 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:31.540028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:31.540002 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs"] Oct 30 06:14:31.540536 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:14:31.540511 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod186bbd33_30f6_4334_b87a_fa151cc6cd61.slice/crio-fc789944b432b7c4a0bc2ca4461999391f218baa00c6d4353fe9c161b0c8d29e WatchSource:0}: Error finding container fc789944b432b7c4a0bc2ca4461999391f218baa00c6d4353fe9c161b0c8d29e: Status 404 returned error can't find the container with id fc789944b432b7c4a0bc2ca4461999391f218baa00c6d4353fe9c161b0c8d29e Oct 30 06:14:32.114687 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:32.114650 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" event={"ID":"186bbd33-30f6-4334-b87a-fa151cc6cd61","Type":"ContainerStarted","Data":"121cfe3d00fb55bab111dbb5f1b9178515ca966aed7281d6cadb05c9c7df82cd"} Oct 30 06:14:32.115099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:32.114710 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" event={"ID":"186bbd33-30f6-4334-b87a-fa151cc6cd61","Type":"ContainerStarted","Data":"fc789944b432b7c4a0bc2ca4461999391f218baa00c6d4353fe9c161b0c8d29e"} Oct 30 06:14:32.115099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:32.114769 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:32.158337 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:32.158274 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" podStartSLOduration=1.779414566 podStartE2EDuration="2.158255973s" totalImagesPullingTime="378.841407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:32.157797397 +0000 UTC m=+1238.666817283" watchObservedRunningTime="2026-10-30 06:14:32.158255973 +0000 UTC m=+1238.667275859" Oct 30 06:14:33.050899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:33.050861 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:33.230993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:33.230958 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:33.372879 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:33.372847 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:34.144578 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:14:34.144540 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:14:34.828418 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:34.828387 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:36.239658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.239628 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz"] Oct 30 06:14:36.240184 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.239884 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" podUID="55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81" containerName="maas-api" containerID="cri-o://5c68d1d1578cf05d6ce235797e167e6ead04a5ca68013eeef3cb0983b984db81" gracePeriod=30 Oct 30 06:14:36.336801 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.336767 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:36.588702 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.588377 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:14:36.616837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.616804 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-metrics-certs\") pod \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " Oct 30 06:14:36.617049 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.616883 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-kube-api-access-jmpb4\" (UniqueName: \"kubernetes.io/projected/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-kube-api-access-jmpb4\") pod \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " Oct 30 06:14:36.617049 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.616956 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-maas-api-tls\") pod \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " Oct 30 06:14:36.617049 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.617030 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-openshift-service-ca\") pod \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\" (UID: \"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81\") " Oct 30 06:14:36.618001 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.617976 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-openshift-service-ca" pod "55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81" (UID: "55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:36.623229 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.621668 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-metrics-certs" pod "55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81" (UID: "55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:36.624355 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.624327 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-kube-api-access-jmpb4" pod "55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81" (UID: "55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81"). InnerVolumeSpecName "kube-api-access-jmpb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:36.624546 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.624519 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-maas-api-tls" pod "55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81" (UID: "55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:36.719199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.719153 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:14:36.719399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.719203 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmpb4\" (UniqueName: \"kubernetes.io/projected/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-kube-api-access-jmpb4\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:14:36.719399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.719240 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:14:36.719399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.719256 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:14:36.874746 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:36.874716 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:37.043929 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:37.043900 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:37.157934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:37.157842 6999 generic.go:356] "Generic (PLEG): container finished" podID="55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81" containerID="5c68d1d1578cf05d6ce235797e167e6ead04a5ca68013eeef3cb0983b984db81" exitCode=0 Oct 30 06:14:37.157934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:37.157893 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" event={"ID":"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81","Type":"ContainerDied","Data":"5c68d1d1578cf05d6ce235797e167e6ead04a5ca68013eeef3cb0983b984db81"} Oct 30 06:14:37.157934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:37.157904 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" Oct 30 06:14:37.157934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:37.157936 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz" event={"ID":"55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81","Type":"ContainerDied","Data":"acd564beb035e5cf3de02b9657258723f97920db5e2fec69b27d54cd5f9d4228"} Oct 30 06:14:37.158313 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:37.157958 6999 scope.go:122] "RemoveContainer" containerID="5c68d1d1578cf05d6ce235797e167e6ead04a5ca68013eeef3cb0983b984db81" Oct 30 06:14:37.171784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:37.171762 6999 scope.go:122] "RemoveContainer" containerID="5c68d1d1578cf05d6ce235797e167e6ead04a5ca68013eeef3cb0983b984db81" Oct 30 06:14:37.172129 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:14:37.172109 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c68d1d1578cf05d6ce235797e167e6ead04a5ca68013eeef3cb0983b984db81\": container with ID starting with 5c68d1d1578cf05d6ce235797e167e6ead04a5ca68013eeef3cb0983b984db81 not found: ID does not exist" containerID="5c68d1d1578cf05d6ce235797e167e6ead04a5ca68013eeef3cb0983b984db81" Oct 30 06:14:37.172248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:37.172140 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c68d1d1578cf05d6ce235797e167e6ead04a5ca68013eeef3cb0983b984db81"} err="failed to get container status \"5c68d1d1578cf05d6ce235797e167e6ead04a5ca68013eeef3cb0983b984db81\": rpc error: code = NotFound desc = could not find container \"5c68d1d1578cf05d6ce235797e167e6ead04a5ca68013eeef3cb0983b984db81\": container with ID starting with 5c68d1d1578cf05d6ce235797e167e6ead04a5ca68013eeef3cb0983b984db81 not found: ID does not exist" Oct 30 06:14:37.262919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:37.262528 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz"] Oct 30 06:14:37.267247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:37.264252 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:37.273716 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:37.273688 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-8tpsz"] Oct 30 06:14:37.609246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:37.609205 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81" path="/var/lib/kubelet/pods/55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81/volumes" Oct 30 06:14:38.088130 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:38.088056 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" Oct 30 06:14:38.131234 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:38.131179 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:14:39.076911 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:39.076882 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:39.242915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:39.242890 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:40.834888 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.834856 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj"] Oct 30 06:14:40.835497 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.835483 6999 cpu_manager.go:414] "containerMap: removing container" podUID="55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81" containerName="maas-api" Oct 30 06:14:40.835568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.835498 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81" containerName="maas-api" Oct 30 06:14:40.835631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.835622 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="55bbb9cb-2f0f-4a8d-ab25-e707a48d0f81" containerName="maas-api" Oct 30 06:14:40.839119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.839104 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:40.870827 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.870790 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c15d3a-serving-cert\"" Oct 30 06:14:40.874436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.874409 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj"] Oct 30 06:14:40.897768 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.897368 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljkb\" (UniqueName: \"kubernetes.io/projected/85f3764f-43fd-4956-b4de-c73fc1285683-kube-api-access-8ljkb\") pod \"maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:40.897768 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.897454 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-maas-api-tls\") pod \"maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:40.897768 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.897580 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f3764f-43fd-4956-b4de-c73fc1285683-openshift-service-ca\") pod \"maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:40.897768 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.897619 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-metrics-certs\") pod \"maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:40.966162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.966134 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:40.999321 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.999290 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f3764f-43fd-4956-b4de-c73fc1285683-openshift-service-ca\") pod \"maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:40.999321 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.999324 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-metrics-certs\") pod \"maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:40.999511 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.999484 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljkb\" (UniqueName: \"kubernetes.io/projected/85f3764f-43fd-4956-b4de-c73fc1285683-kube-api-access-8ljkb\") pod \"maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:40.999511 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:40.999505 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-maas-api-tls\") pod \"maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:40.999636 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:14:40.999622 6999 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-serving-cert: secret "maas-api-e2e-api-a-c15d3a-serving-cert" not found Oct 30 06:14:40.999694 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:14:40.999687 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-maas-api-tls podName:85f3764f-43fd-4956-b4de-c73fc1285683 nodeName:}" failed. No retries permitted until 2026-10-30 06:14:41.499668027 +0000 UTC m=+1248.008687901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-maas-api-tls") pod "maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" (UID: "85f3764f-43fd-4956-b4de-c73fc1285683") : secret "maas-api-e2e-api-a-c15d3a-serving-cert" not found Oct 30 06:14:41.000070 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:41.000048 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f3764f-43fd-4956-b4de-c73fc1285683-openshift-service-ca\") pod \"maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:41.002010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:41.001988 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-metrics-certs\") pod \"maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:41.024626 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:41.024593 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljkb\" (UniqueName: \"kubernetes.io/projected/85f3764f-43fd-4956-b4de-c73fc1285683-kube-api-access-8ljkb\") pod \"maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:41.120502 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:41.120472 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:41.263432 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:41.263400 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:41.493416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:41.493330 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:41.507480 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:41.507433 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-maas-api-tls\") pod \"maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:41.512296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:41.512157 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-maas-api-tls\") pod \"maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:41.752740 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:41.752652 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:42.273283 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:14:42.273252 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f3764f_43fd_4956_b4de_c73fc1285683.slice/crio-daa9b4c95f58f182c33af67260f621be73574f16366ee69a233001d81cf0aa4d WatchSource:0}: Error finding container daa9b4c95f58f182c33af67260f621be73574f16366ee69a233001d81cf0aa4d: Status 404 returned error can't find the container with id daa9b4c95f58f182c33af67260f621be73574f16366ee69a233001d81cf0aa4d Oct 30 06:14:42.282152 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:42.282123 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj"] Oct 30 06:14:43.212360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:43.212322 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" event={"ID":"85f3764f-43fd-4956-b4de-c73fc1285683","Type":"ContainerStarted","Data":"319a801d9e3782bf40c1e54397a76a6c0d2053d571673f9954aee6e2667ad91c"} Oct 30 06:14:43.212583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:43.212389 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" event={"ID":"85f3764f-43fd-4956-b4de-c73fc1285683","Type":"ContainerStarted","Data":"daa9b4c95f58f182c33af67260f621be73574f16366ee69a233001d81cf0aa4d"} Oct 30 06:14:43.212583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:43.212433 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:43.269631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:43.269579 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" podStartSLOduration=2.942948345 podStartE2EDuration="3.269564181s" totalImagesPullingTime="326.615836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:43.266593325 +0000 UTC m=+1249.775613214" watchObservedRunningTime="2026-10-30 06:14:43.269564181 +0000 UTC m=+1249.778584118" Oct 30 06:14:43.364973 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:43.364940 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:45.275054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:45.274994 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:47.261068 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:47.261034 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:49.255555 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:49.255523 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:14:49.728236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:49.727271 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:50.000299 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:49.994681 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:50.252300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:50.251007 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:50.888022 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:50.887988 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:51.047604 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:51.047563 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:52.257099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:52.257063 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:54.269570 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:54.269478 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:55.219089 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:55.219035 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:56.164262 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.164229 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff"] Oct 30 06:14:56.170341 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.170325 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:14:56.192501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.192479 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff"] Oct 30 06:14:56.200432 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.200408 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ba44b248-serving-cert\"" Oct 30 06:14:56.285450 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.285389 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12365d0d-c44f-488e-9362-d5f74d4de585-metrics-certs\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:14:56.285685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.285638 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12365d0d-c44f-488e-9362-d5f74d4de585-maas-api-tls\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:14:56.285768 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.285684 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnsj4\" (UniqueName: \"kubernetes.io/projected/12365d0d-c44f-488e-9362-d5f74d4de585-kube-api-access-mnsj4\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:14:56.285837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.285814 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12365d0d-c44f-488e-9362-d5f74d4de585-openshift-service-ca\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:14:56.356662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.356632 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:56.386921 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.386889 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnsj4\" (UniqueName: \"kubernetes.io/projected/12365d0d-c44f-488e-9362-d5f74d4de585-kube-api-access-mnsj4\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:14:56.387051 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.386940 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12365d0d-c44f-488e-9362-d5f74d4de585-openshift-service-ca\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:14:56.387100 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.387058 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12365d0d-c44f-488e-9362-d5f74d4de585-metrics-certs\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:14:56.387173 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.387156 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12365d0d-c44f-488e-9362-d5f74d4de585-maas-api-tls\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:14:56.387771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.387746 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12365d0d-c44f-488e-9362-d5f74d4de585-openshift-service-ca\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:14:56.390073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.390026 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12365d0d-c44f-488e-9362-d5f74d4de585-maas-api-tls\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:14:56.392538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.392516 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12365d0d-c44f-488e-9362-d5f74d4de585-metrics-certs\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:14:56.417132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.417057 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnsj4\" (UniqueName: \"kubernetes.io/projected/12365d0d-c44f-488e-9362-d5f74d4de585-kube-api-access-mnsj4\") pod \"maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:14:56.484425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.484391 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:14:56.671437 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.671355 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:56.822311 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:56.822284 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff"] Oct 30 06:14:56.823976 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:14:56.823950 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12365d0d_c44f_488e_9362_d5f74d4de585.slice/crio-c238288a2b95c9bc831249b7fb21b4840df512c619590f4fb6e1c733c19eb475 WatchSource:0}: Error finding container c238288a2b95c9bc831249b7fb21b4840df512c619590f4fb6e1c733c19eb475: Status 404 returned error can't find the container with id c238288a2b95c9bc831249b7fb21b4840df512c619590f4fb6e1c733c19eb475 Oct 30 06:14:57.388776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:57.388725 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" event={"ID":"12365d0d-c44f-488e-9362-d5f74d4de585","Type":"ContainerStarted","Data":"6eead7fa7571004f80711472bf8549155abf2a85c0157191d222b25b9786e16e"} Oct 30 06:14:57.389279 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:57.388834 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" event={"ID":"12365d0d-c44f-488e-9362-d5f74d4de585","Type":"ContainerStarted","Data":"c238288a2b95c9bc831249b7fb21b4840df512c619590f4fb6e1c733c19eb475"} Oct 30 06:14:57.389922 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:57.389897 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:14:57.432475 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:57.432426 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" podStartSLOduration=1.063528643 podStartE2EDuration="1.432410988s" totalImagesPullingTime="368.882345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:57.431383268 +0000 UTC m=+1263.940403156" watchObservedRunningTime="2026-10-30 06:14:57.432410988 +0000 UTC m=+1263.941430893" Oct 30 06:14:58.548807 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:58.548771 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:14:59.301105 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:14:59.301070 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:00.183876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.183845 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67"] Oct 30 06:15:00.188200 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.188177 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" Oct 30 06:15:00.209060 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.209043 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:15:00.234255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.228020 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67"] Oct 30 06:15:00.234255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.233582 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj"] Oct 30 06:15:00.238878 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.238857 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" Oct 30 06:15:00.240155 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.240002 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dbc792-be6b-4283-a38b-31c945b7a537-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67\" (UID: \"98dbc792-be6b-4283-a38b-31c945b7a537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" Oct 30 06:15:00.240375 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.240254 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4l64\" (UniqueName: \"kubernetes.io/projected/98dbc792-be6b-4283-a38b-31c945b7a537-kube-api-access-d4l64\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67\" (UID: \"98dbc792-be6b-4283-a38b-31c945b7a537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" Oct 30 06:15:00.247292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.244936 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4"] Oct 30 06:15:00.250322 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.250299 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" Oct 30 06:15:00.261531 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.261505 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj"] Oct 30 06:15:00.282255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.282202 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4"] Oct 30 06:15:00.341319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.341286 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b76f03-12c8-4308-b2c7-f5261158504d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj\" (UID: \"57b76f03-12c8-4308-b2c7-f5261158504d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" Oct 30 06:15:00.341499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.341345 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkwsg\" (UniqueName: \"kubernetes.io/projected/57b76f03-12c8-4308-b2c7-f5261158504d-kube-api-access-tkwsg\") pod \"maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj\" (UID: \"57b76f03-12c8-4308-b2c7-f5261158504d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" Oct 30 06:15:00.341499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.341480 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dbc792-be6b-4283-a38b-31c945b7a537-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67\" (UID: \"98dbc792-be6b-4283-a38b-31c945b7a537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" Oct 30 06:15:00.341574 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.341529 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a11f681-c9a2-4d78-b6ac-857b4e7d8337-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4\" (UID: \"0a11f681-c9a2-4d78-b6ac-857b4e7d8337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" Oct 30 06:15:00.341574 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.341557 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4l64\" (UniqueName: \"kubernetes.io/projected/98dbc792-be6b-4283-a38b-31c945b7a537-kube-api-access-d4l64\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67\" (UID: \"98dbc792-be6b-4283-a38b-31c945b7a537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" Oct 30 06:15:00.341640 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.341574 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4rn6\" (UniqueName: \"kubernetes.io/projected/0a11f681-c9a2-4d78-b6ac-857b4e7d8337-kube-api-access-k4rn6\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4\" (UID: \"0a11f681-c9a2-4d78-b6ac-857b4e7d8337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" Oct 30 06:15:00.342255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.342238 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dbc792-be6b-4283-a38b-31c945b7a537-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67\" (UID: \"98dbc792-be6b-4283-a38b-31c945b7a537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" Oct 30 06:15:00.367121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.367098 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4l64\" (UniqueName: \"kubernetes.io/projected/98dbc792-be6b-4283-a38b-31c945b7a537-kube-api-access-d4l64\") pod \"maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67\" (UID: \"98dbc792-be6b-4283-a38b-31c945b7a537\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" Oct 30 06:15:00.443040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.442955 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a11f681-c9a2-4d78-b6ac-857b4e7d8337-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4\" (UID: \"0a11f681-c9a2-4d78-b6ac-857b4e7d8337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" Oct 30 06:15:00.443040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.442996 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4rn6\" (UniqueName: \"kubernetes.io/projected/0a11f681-c9a2-4d78-b6ac-857b4e7d8337-kube-api-access-k4rn6\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4\" (UID: \"0a11f681-c9a2-4d78-b6ac-857b4e7d8337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" Oct 30 06:15:00.443310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.443157 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b76f03-12c8-4308-b2c7-f5261158504d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj\" (UID: \"57b76f03-12c8-4308-b2c7-f5261158504d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" Oct 30 06:15:00.443310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.443230 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkwsg\" (UniqueName: \"kubernetes.io/projected/57b76f03-12c8-4308-b2c7-f5261158504d-kube-api-access-tkwsg\") pod \"maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj\" (UID: \"57b76f03-12c8-4308-b2c7-f5261158504d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" Oct 30 06:15:00.443775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.443752 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a11f681-c9a2-4d78-b6ac-857b4e7d8337-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4\" (UID: \"0a11f681-c9a2-4d78-b6ac-857b4e7d8337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" Oct 30 06:15:00.443862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.443846 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b76f03-12c8-4308-b2c7-f5261158504d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj\" (UID: \"57b76f03-12c8-4308-b2c7-f5261158504d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" Oct 30 06:15:00.466579 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.466546 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4rn6\" (UniqueName: \"kubernetes.io/projected/0a11f681-c9a2-4d78-b6ac-857b4e7d8337-kube-api-access-k4rn6\") pod \"maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4\" (UID: \"0a11f681-c9a2-4d78-b6ac-857b4e7d8337\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" Oct 30 06:15:00.471814 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.471791 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkwsg\" (UniqueName: \"kubernetes.io/projected/57b76f03-12c8-4308-b2c7-f5261158504d-kube-api-access-tkwsg\") pod \"maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj\" (UID: \"57b76f03-12c8-4308-b2c7-f5261158504d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" Oct 30 06:15:00.504090 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.504050 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" Oct 30 06:15:00.565005 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.564971 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" Oct 30 06:15:00.574591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.572019 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:00.583126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.583088 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" Oct 30 06:15:00.815616 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.815589 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67"] Oct 30 06:15:00.816626 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:15:00.816592 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98dbc792_be6b_4283_a38b_31c945b7a537.slice/crio-86670d990d557fd5e0d5bf219f9c9770bb95f397c7eeb39babf2a365280b8827 WatchSource:0}: Error finding container 86670d990d557fd5e0d5bf219f9c9770bb95f397c7eeb39babf2a365280b8827: Status 404 returned error can't find the container with id 86670d990d557fd5e0d5bf219f9c9770bb95f397c7eeb39babf2a365280b8827 Oct 30 06:15:00.910270 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.910242 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj"] Oct 30 06:15:00.910400 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:15:00.910327 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57b76f03_12c8_4308_b2c7_f5261158504d.slice/crio-9d344b9607472a03777213a8471888978a14ff135842cd8fdd326d2f3d4f311e WatchSource:0}: Error finding container 9d344b9607472a03777213a8471888978a14ff135842cd8fdd326d2f3d4f311e: Status 404 returned error can't find the container with id 9d344b9607472a03777213a8471888978a14ff135842cd8fdd326d2f3d4f311e Oct 30 06:15:00.930906 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:00.930880 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4"] Oct 30 06:15:00.931139 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:15:00.931113 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a11f681_c9a2_4d78_b6ac_857b4e7d8337.slice/crio-9737174155bce9b42449045aa53566603e6d13fb4ba344e6a7915f0bebbdf2ef WatchSource:0}: Error finding container 9737174155bce9b42449045aa53566603e6d13fb4ba344e6a7915f0bebbdf2ef: Status 404 returned error can't find the container with id 9737174155bce9b42449045aa53566603e6d13fb4ba344e6a7915f0bebbdf2ef Oct 30 06:15:01.172798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:01.172777 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:01.416867 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:01.416834 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" event={"ID":"98dbc792-be6b-4283-a38b-31c945b7a537","Type":"ContainerStarted","Data":"966ee385897b5e9c1954d797803723ce85ed2ae77e1c7cd525efa7a5719810dd"} Oct 30 06:15:01.417370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:01.416901 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" event={"ID":"98dbc792-be6b-4283-a38b-31c945b7a537","Type":"ContainerStarted","Data":"86670d990d557fd5e0d5bf219f9c9770bb95f397c7eeb39babf2a365280b8827"} Oct 30 06:15:01.418601 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:01.418573 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" event={"ID":"0a11f681-c9a2-4d78-b6ac-857b4e7d8337","Type":"ContainerStarted","Data":"ce663121d36cac408970ca8151813f000766a98caa785b9ffb5c96d909ef2248"} Oct 30 06:15:01.418741 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:01.418622 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" event={"ID":"0a11f681-c9a2-4d78-b6ac-857b4e7d8337","Type":"ContainerStarted","Data":"9737174155bce9b42449045aa53566603e6d13fb4ba344e6a7915f0bebbdf2ef"} Oct 30 06:15:01.420256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:01.420233 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" event={"ID":"57b76f03-12c8-4308-b2c7-f5261158504d","Type":"ContainerStarted","Data":"5f9734627a701d9e7bc576a46aee7ade0b4688a7786843306631c2c75ab378ee"} Oct 30 06:15:01.420388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:01.420271 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" event={"ID":"57b76f03-12c8-4308-b2c7-f5261158504d","Type":"ContainerStarted","Data":"9d344b9607472a03777213a8471888978a14ff135842cd8fdd326d2f3d4f311e"} Oct 30 06:15:01.460964 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:01.460920 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" podStartSLOduration=1.460908463 podStartE2EDuration="1.460908463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:01.457296588 +0000 UTC m=+1267.966316475" watchObservedRunningTime="2026-10-30 06:15:01.460908463 +0000 UTC m=+1267.969928347" Oct 30 06:15:01.498952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:01.498906 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" podStartSLOduration=1.498892783 podStartE2EDuration="1.498892783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:01.496471607 +0000 UTC m=+1268.005491506" watchObservedRunningTime="2026-10-30 06:15:01.498892783 +0000 UTC m=+1268.007912671" Oct 30 06:15:01.537004 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:01.536958 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" podStartSLOduration=1.536945437 podStartE2EDuration="1.536945437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:01.535411207 +0000 UTC m=+1268.044431094" watchObservedRunningTime="2026-10-30 06:15:01.536945437 +0000 UTC m=+1268.045965326" Oct 30 06:15:02.428539 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:02.428505 6999 generic.go:356] "Generic (PLEG): container finished" podID="57b76f03-12c8-4308-b2c7-f5261158504d" containerID="5f9734627a701d9e7bc576a46aee7ade0b4688a7786843306631c2c75ab378ee" exitCode=60 Oct 30 06:15:02.429072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:02.428589 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" event={"ID":"57b76f03-12c8-4308-b2c7-f5261158504d","Type":"ContainerDied","Data":"5f9734627a701d9e7bc576a46aee7ade0b4688a7786843306631c2c75ab378ee"} Oct 30 06:15:02.429072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:02.428907 6999 scope.go:122] "RemoveContainer" containerID="5f9734627a701d9e7bc576a46aee7ade0b4688a7786843306631c2c75ab378ee" Oct 30 06:15:02.430427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:02.430401 6999 generic.go:356] "Generic (PLEG): container finished" podID="98dbc792-be6b-4283-a38b-31c945b7a537" containerID="966ee385897b5e9c1954d797803723ce85ed2ae77e1c7cd525efa7a5719810dd" exitCode=60 Oct 30 06:15:02.430548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:02.430505 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" event={"ID":"98dbc792-be6b-4283-a38b-31c945b7a537","Type":"ContainerDied","Data":"966ee385897b5e9c1954d797803723ce85ed2ae77e1c7cd525efa7a5719810dd"} Oct 30 06:15:02.430735 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:02.430716 6999 scope.go:122] "RemoveContainer" containerID="966ee385897b5e9c1954d797803723ce85ed2ae77e1c7cd525efa7a5719810dd" Oct 30 06:15:02.433592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:02.433572 6999 generic.go:356] "Generic (PLEG): container finished" podID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" containerID="ce663121d36cac408970ca8151813f000766a98caa785b9ffb5c96d909ef2248" exitCode=60 Oct 30 06:15:02.433678 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:02.433623 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" event={"ID":"0a11f681-c9a2-4d78-b6ac-857b4e7d8337","Type":"ContainerDied","Data":"ce663121d36cac408970ca8151813f000766a98caa785b9ffb5c96d909ef2248"} Oct 30 06:15:02.433896 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:02.433875 6999 scope.go:122] "RemoveContainer" containerID="ce663121d36cac408970ca8151813f000766a98caa785b9ffb5c96d909ef2248" Oct 30 06:15:02.444359 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:02.443789 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:03.443268 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:03.443156 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" event={"ID":"98dbc792-be6b-4283-a38b-31c945b7a537","Type":"ContainerStarted","Data":"ed5c70f066135dded3fd14c49cacbf2d37e6c59371bc02e661194ea54c87ee33"} Oct 30 06:15:03.445302 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:03.445267 6999 generic.go:356] "Generic (PLEG): container finished" podID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" containerID="bf9c8f399190f26d92b80a8690db04d0f18b5a9ee87d5fe413ec1ac0e6154409" exitCode=60 Oct 30 06:15:03.445440 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:03.445389 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" event={"ID":"0a11f681-c9a2-4d78-b6ac-857b4e7d8337","Type":"ContainerDied","Data":"bf9c8f399190f26d92b80a8690db04d0f18b5a9ee87d5fe413ec1ac0e6154409"} Oct 30 06:15:03.445495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:03.445451 6999 scope.go:122] "RemoveContainer" containerID="ce663121d36cac408970ca8151813f000766a98caa785b9ffb5c96d909ef2248" Oct 30 06:15:03.445679 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:03.445656 6999 scope.go:122] "RemoveContainer" containerID="bf9c8f399190f26d92b80a8690db04d0f18b5a9ee87d5fe413ec1ac0e6154409" Oct 30 06:15:03.446003 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:03.445982 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4_odh-ai-gateway-infra(0a11f681-c9a2-4d78-b6ac-857b4e7d8337)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" podUID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" Oct 30 06:15:03.447499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:03.447478 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" event={"ID":"57b76f03-12c8-4308-b2c7-f5261158504d","Type":"ContainerStarted","Data":"0b981a79be6ff2c8e92865eb3bfbb072ed3cdfce8820107cdda0c584487eb254"} Oct 30 06:15:04.151822 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:04.151780 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:15:04.407631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:04.407554 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:15:04.458854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:04.458815 6999 generic.go:356] "Generic (PLEG): container finished" podID="98dbc792-be6b-4283-a38b-31c945b7a537" containerID="ed5c70f066135dded3fd14c49cacbf2d37e6c59371bc02e661194ea54c87ee33" exitCode=60 Oct 30 06:15:04.459367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:04.458959 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" event={"ID":"98dbc792-be6b-4283-a38b-31c945b7a537","Type":"ContainerDied","Data":"ed5c70f066135dded3fd14c49cacbf2d37e6c59371bc02e661194ea54c87ee33"} Oct 30 06:15:04.459367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:04.459031 6999 scope.go:122] "RemoveContainer" containerID="966ee385897b5e9c1954d797803723ce85ed2ae77e1c7cd525efa7a5719810dd" Oct 30 06:15:04.459439 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:04.459373 6999 scope.go:122] "RemoveContainer" containerID="ed5c70f066135dded3fd14c49cacbf2d37e6c59371bc02e661194ea54c87ee33" Oct 30 06:15:04.459744 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:04.459717 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67_odh-ai-gateway-infra(98dbc792-be6b-4283-a38b-31c945b7a537)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" podUID="98dbc792-be6b-4283-a38b-31c945b7a537" Oct 30 06:15:04.461835 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:04.461816 6999 scope.go:122] "RemoveContainer" containerID="bf9c8f399190f26d92b80a8690db04d0f18b5a9ee87d5fe413ec1ac0e6154409" Oct 30 06:15:04.462161 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:04.462131 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4_odh-ai-gateway-infra(0a11f681-c9a2-4d78-b6ac-857b4e7d8337)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" podUID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" Oct 30 06:15:04.463674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:04.463648 6999 generic.go:356] "Generic (PLEG): container finished" podID="57b76f03-12c8-4308-b2c7-f5261158504d" containerID="0b981a79be6ff2c8e92865eb3bfbb072ed3cdfce8820107cdda0c584487eb254" exitCode=60 Oct 30 06:15:04.463847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:04.463733 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" event={"ID":"57b76f03-12c8-4308-b2c7-f5261158504d","Type":"ContainerDied","Data":"0b981a79be6ff2c8e92865eb3bfbb072ed3cdfce8820107cdda0c584487eb254"} Oct 30 06:15:04.463989 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:04.463972 6999 scope.go:122] "RemoveContainer" containerID="0b981a79be6ff2c8e92865eb3bfbb072ed3cdfce8820107cdda0c584487eb254" Oct 30 06:15:04.464292 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:04.464266 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj_odh-ai-gateway-infra(57b76f03-12c8-4308-b2c7-f5261158504d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" podUID="57b76f03-12c8-4308-b2c7-f5261158504d" Oct 30 06:15:04.478187 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:04.478159 6999 scope.go:122] "RemoveContainer" containerID="5f9734627a701d9e7bc576a46aee7ade0b4688a7786843306631c2c75ab378ee" Oct 30 06:15:04.496278 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:04.495951 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:05.472798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:05.472773 6999 scope.go:122] "RemoveContainer" containerID="0b981a79be6ff2c8e92865eb3bfbb072ed3cdfce8820107cdda0c584487eb254" Oct 30 06:15:05.473267 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:05.473155 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj_odh-ai-gateway-infra(57b76f03-12c8-4308-b2c7-f5261158504d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" podUID="57b76f03-12c8-4308-b2c7-f5261158504d" Oct 30 06:15:05.474542 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:05.474525 6999 scope.go:122] "RemoveContainer" containerID="ed5c70f066135dded3fd14c49cacbf2d37e6c59371bc02e661194ea54c87ee33" Oct 30 06:15:05.474793 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:05.474775 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67_odh-ai-gateway-infra(98dbc792-be6b-4283-a38b-31c945b7a537)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" podUID="98dbc792-be6b-4283-a38b-31c945b7a537" Oct 30 06:15:05.542331 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:05.542298 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:05.750576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:05.750474 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:06.657624 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:06.657582 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:07.143746 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:07.143710 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw"] Oct 30 06:15:07.320882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:07.320850 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:07.473848 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:07.473757 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:08.486794 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:08.486768 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:09.656720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:09.656669 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:09.910872 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:09.910792 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph"] Oct 30 06:15:09.911057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:09.911033 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" podUID="4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6" containerName="maas-api" containerID="cri-o://0e249a7958506e4147d91a58e984d7bcc1cf3367e339c2ce65cd37cf2ac63d23" gracePeriod=30 Oct 30 06:15:10.128766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.128573 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:10.221770 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.221732 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:15:10.389238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.389192 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-maas-api-tls\") pod \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " Oct 30 06:15:10.389506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.389305 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-metrics-certs\") pod \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " Oct 30 06:15:10.389506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.389460 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-kube-api-access-2tfw7\" (UniqueName: \"kubernetes.io/projected/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-kube-api-access-2tfw7\") pod \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " Oct 30 06:15:10.389638 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.389507 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-openshift-service-ca\") pod \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\" (UID: \"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6\") " Oct 30 06:15:10.389996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.389954 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-openshift-service-ca" pod "4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6" (UID: "4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:10.390675 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.390651 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:10.393246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.391484 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:10.393246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.393109 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-kube-api-access-2tfw7" pod "4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6" (UID: "4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6"). InnerVolumeSpecName "kube-api-access-2tfw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:10.393246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.393199 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-maas-api-tls" pod "4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6" (UID: "4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:15:10.393418 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.393296 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-metrics-certs" pod "4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6" (UID: "4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:15:10.492966 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.492864 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:10.492966 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.492897 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:10.492966 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.492906 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tfw7\" (UniqueName: \"kubernetes.io/projected/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6-kube-api-access-2tfw7\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:10.514968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.514886 6999 generic.go:356] "Generic (PLEG): container finished" podID="4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6" containerID="0e249a7958506e4147d91a58e984d7bcc1cf3367e339c2ce65cd37cf2ac63d23" exitCode=0 Oct 30 06:15:10.514968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.514949 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" Oct 30 06:15:10.515178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.515101 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" event={"ID":"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6","Type":"ContainerDied","Data":"0e249a7958506e4147d91a58e984d7bcc1cf3367e339c2ce65cd37cf2ac63d23"} Oct 30 06:15:10.515178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.515142 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph" event={"ID":"4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6","Type":"ContainerDied","Data":"4c4faa0c49409634dbb019538833a5f60395ef460918cca872a1a06606da2a3c"} Oct 30 06:15:10.515178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.515165 6999 scope.go:122] "RemoveContainer" containerID="0e249a7958506e4147d91a58e984d7bcc1cf3367e339c2ce65cd37cf2ac63d23" Oct 30 06:15:10.529527 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.529501 6999 scope.go:122] "RemoveContainer" containerID="0e249a7958506e4147d91a58e984d7bcc1cf3367e339c2ce65cd37cf2ac63d23" Oct 30 06:15:10.529835 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:10.529811 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e249a7958506e4147d91a58e984d7bcc1cf3367e339c2ce65cd37cf2ac63d23\": container with ID starting with 0e249a7958506e4147d91a58e984d7bcc1cf3367e339c2ce65cd37cf2ac63d23 not found: ID does not exist" containerID="0e249a7958506e4147d91a58e984d7bcc1cf3367e339c2ce65cd37cf2ac63d23" Oct 30 06:15:10.529902 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.529849 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e249a7958506e4147d91a58e984d7bcc1cf3367e339c2ce65cd37cf2ac63d23"} err="failed to get container status \"0e249a7958506e4147d91a58e984d7bcc1cf3367e339c2ce65cd37cf2ac63d23\": rpc error: code = NotFound desc = could not find container \"0e249a7958506e4147d91a58e984d7bcc1cf3367e339c2ce65cd37cf2ac63d23\": container with ID starting with 0e249a7958506e4147d91a58e984d7bcc1cf3367e339c2ce65cd37cf2ac63d23 not found: ID does not exist" Oct 30 06:15:10.573940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.573905 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph"] Oct 30 06:15:10.589404 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.589378 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f789d7-5465df4f74-bgrph"] Oct 30 06:15:10.663130 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.663085 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:10.823181 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:10.823071 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:11.390950 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:11.390922 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:11.548370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:11.548327 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:11.610764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:11.610709 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6" path="/var/lib/kubelet/pods/4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6/volumes" Oct 30 06:15:11.696416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:11.696270 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:12.145050 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.145012 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" podUID="6815f024-a76f-45e1-b16d-c1d3ed0921b3" containerName="payload-processing" containerID="cri-o://50dd2b211b0a5b3afc88fdb0f6c2e414096c01a17e1cfb9be09d39cb4e560c28" gracePeriod=25 Oct 30 06:15:12.424551 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.424526 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" Oct 30 06:15:12.525314 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.525278 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6815f024-a76f-45e1-b16d-c1d3ed0921b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6815f024-a76f-45e1-b16d-c1d3ed0921b3-plugins-config-volume\") pod \"6815f024-a76f-45e1-b16d-c1d3ed0921b3\" (UID: \"6815f024-a76f-45e1-b16d-c1d3ed0921b3\") " Oct 30 06:15:12.525508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.525354 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6815f024-a76f-45e1-b16d-c1d3ed0921b3-kube-api-access-bcsmb\" (UniqueName: \"kubernetes.io/projected/6815f024-a76f-45e1-b16d-c1d3ed0921b3-kube-api-access-bcsmb\") pod \"6815f024-a76f-45e1-b16d-c1d3ed0921b3\" (UID: \"6815f024-a76f-45e1-b16d-c1d3ed0921b3\") " Oct 30 06:15:12.525724 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.525698 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6815f024-a76f-45e1-b16d-c1d3ed0921b3-plugins-config-volume" pod "6815f024-a76f-45e1-b16d-c1d3ed0921b3" (UID: "6815f024-a76f-45e1-b16d-c1d3ed0921b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:12.526120 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.526068 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6815f024-a76f-45e1-b16d-c1d3ed0921b3-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:12.527722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.527700 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6815f024-a76f-45e1-b16d-c1d3ed0921b3-kube-api-access-bcsmb" pod "6815f024-a76f-45e1-b16d-c1d3ed0921b3" (UID: "6815f024-a76f-45e1-b16d-c1d3ed0921b3"). InnerVolumeSpecName "kube-api-access-bcsmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:12.536425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.536394 6999 generic.go:356] "Generic (PLEG): container finished" podID="6815f024-a76f-45e1-b16d-c1d3ed0921b3" containerID="50dd2b211b0a5b3afc88fdb0f6c2e414096c01a17e1cfb9be09d39cb4e560c28" exitCode=0 Oct 30 06:15:12.536515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.536439 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" event={"ID":"6815f024-a76f-45e1-b16d-c1d3ed0921b3","Type":"ContainerDied","Data":"50dd2b211b0a5b3afc88fdb0f6c2e414096c01a17e1cfb9be09d39cb4e560c28"} Oct 30 06:15:12.536515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.536460 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" Oct 30 06:15:12.536515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.536484 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw" event={"ID":"6815f024-a76f-45e1-b16d-c1d3ed0921b3","Type":"ContainerDied","Data":"843cd0c335eda0a643dd35e447b4bff1bb8d178deade07c608611a390c03cc6c"} Oct 30 06:15:12.536515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.536504 6999 scope.go:122] "RemoveContainer" containerID="50dd2b211b0a5b3afc88fdb0f6c2e414096c01a17e1cfb9be09d39cb4e560c28" Oct 30 06:15:12.547080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.547055 6999 scope.go:122] "RemoveContainer" containerID="50dd2b211b0a5b3afc88fdb0f6c2e414096c01a17e1cfb9be09d39cb4e560c28" Oct 30 06:15:12.547406 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:12.547383 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50dd2b211b0a5b3afc88fdb0f6c2e414096c01a17e1cfb9be09d39cb4e560c28\": container with ID starting with 50dd2b211b0a5b3afc88fdb0f6c2e414096c01a17e1cfb9be09d39cb4e560c28 not found: ID does not exist" containerID="50dd2b211b0a5b3afc88fdb0f6c2e414096c01a17e1cfb9be09d39cb4e560c28" Oct 30 06:15:12.547482 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.547413 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50dd2b211b0a5b3afc88fdb0f6c2e414096c01a17e1cfb9be09d39cb4e560c28"} err="failed to get container status \"50dd2b211b0a5b3afc88fdb0f6c2e414096c01a17e1cfb9be09d39cb4e560c28\": rpc error: code = NotFound desc = could not find container \"50dd2b211b0a5b3afc88fdb0f6c2e414096c01a17e1cfb9be09d39cb4e560c28\": container with ID starting with 50dd2b211b0a5b3afc88fdb0f6c2e414096c01a17e1cfb9be09d39cb4e560c28 not found: ID does not exist" Oct 30 06:15:12.590118 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.590082 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw"] Oct 30 06:15:12.594381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.594354 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e6df62-cd88fc896-qzpvw"] Oct 30 06:15:12.627747 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.627717 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcsmb\" (UniqueName: \"kubernetes.io/projected/6815f024-a76f-45e1-b16d-c1d3ed0921b3-kube-api-access-bcsmb\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:12.934044 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:12.933994 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:13.609062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:13.609030 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6815f024-a76f-45e1-b16d-c1d3ed0921b3" path="/var/lib/kubelet/pods/6815f024-a76f-45e1-b16d-c1d3ed0921b3/volumes" Oct 30 06:15:14.601827 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:14.601799 6999 scope.go:122] "RemoveContainer" containerID="ed5c70f066135dded3fd14c49cacbf2d37e6c59371bc02e661194ea54c87ee33" Oct 30 06:15:14.896061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:14.896024 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:15.384673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:15.384640 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:15.619486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:15.619442 6999 generic.go:356] "Generic (PLEG): container finished" podID="98dbc792-be6b-4283-a38b-31c945b7a537" containerID="4ecb8004c44357043f201da629889f9a18d61f83c8262ad947b691e5b4cee50c" exitCode=60 Oct 30 06:15:15.638641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:15.638549 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:15.669239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:15.664510 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" event={"ID":"98dbc792-be6b-4283-a38b-31c945b7a537","Type":"ContainerDied","Data":"4ecb8004c44357043f201da629889f9a18d61f83c8262ad947b691e5b4cee50c"} Oct 30 06:15:15.669239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:15.664600 6999 scope.go:122] "RemoveContainer" containerID="ed5c70f066135dded3fd14c49cacbf2d37e6c59371bc02e661194ea54c87ee33" Oct 30 06:15:15.669239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:15.665324 6999 scope.go:122] "RemoveContainer" containerID="4ecb8004c44357043f201da629889f9a18d61f83c8262ad947b691e5b4cee50c" Oct 30 06:15:15.669239 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:15.665738 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67_odh-ai-gateway-infra(98dbc792-be6b-4283-a38b-31c945b7a537)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" podUID="98dbc792-be6b-4283-a38b-31c945b7a537" Oct 30 06:15:15.669239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:15.666167 6999 scope.go:122] "RemoveContainer" containerID="0b981a79be6ff2c8e92865eb3bfbb072ed3cdfce8820107cdda0c584487eb254" Oct 30 06:15:16.392398 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:16.392364 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:16.629072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:16.629030 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" event={"ID":"57b76f03-12c8-4308-b2c7-f5261158504d","Type":"ContainerStarted","Data":"590890a1d471fd8a5a71a9be8490dbacc72034a5a2e63191a6b3f77c37155cf7"} Oct 30 06:15:16.688002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:16.687974 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:16.737883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:16.737841 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67"] Oct 30 06:15:16.943892 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:16.943866 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" Oct 30 06:15:17.045132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.045100 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98dbc792-be6b-4283-a38b-31c945b7a537-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dbc792-be6b-4283-a38b-31c945b7a537-maas-api-service-ca\") pod \"98dbc792-be6b-4283-a38b-31c945b7a537\" (UID: \"98dbc792-be6b-4283-a38b-31c945b7a537\") " Oct 30 06:15:17.045576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.045544 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98dbc792-be6b-4283-a38b-31c945b7a537-maas-api-service-ca" pod "98dbc792-be6b-4283-a38b-31c945b7a537" (UID: "98dbc792-be6b-4283-a38b-31c945b7a537"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:17.045705 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.045688 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98dbc792-be6b-4283-a38b-31c945b7a537-kube-api-access-d4l64\" (UniqueName: \"kubernetes.io/projected/98dbc792-be6b-4283-a38b-31c945b7a537-kube-api-access-d4l64\") pod \"98dbc792-be6b-4283-a38b-31c945b7a537\" (UID: \"98dbc792-be6b-4283-a38b-31c945b7a537\") " Oct 30 06:15:17.046655 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.046606 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98dbc792-be6b-4283-a38b-31c945b7a537-maas-api-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:17.048493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.048456 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98dbc792-be6b-4283-a38b-31c945b7a537-kube-api-access-d4l64" pod "98dbc792-be6b-4283-a38b-31c945b7a537" (UID: "98dbc792-be6b-4283-a38b-31c945b7a537"). InnerVolumeSpecName "kube-api-access-d4l64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:17.148421 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.148365 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4l64\" (UniqueName: \"kubernetes.io/projected/98dbc792-be6b-4283-a38b-31c945b7a537-kube-api-access-d4l64\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:17.344905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.344798 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:17.529106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.529066 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw"] Oct 30 06:15:17.530058 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.530036 6999 cpu_manager.go:414] "containerMap: removing container" podUID="98dbc792-be6b-4283-a38b-31c945b7a537" containerName="cleanup" Oct 30 06:15:17.530153 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.530062 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98dbc792-be6b-4283-a38b-31c945b7a537" containerName="cleanup" Oct 30 06:15:17.530153 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.530116 6999 cpu_manager.go:414] "containerMap: removing container" podUID="98dbc792-be6b-4283-a38b-31c945b7a537" containerName="cleanup" Oct 30 06:15:17.530153 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.530124 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98dbc792-be6b-4283-a38b-31c945b7a537" containerName="cleanup" Oct 30 06:15:17.530289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.530170 6999 cpu_manager.go:414] "containerMap: removing container" podUID="4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6" containerName="maas-api" Oct 30 06:15:17.530289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.530178 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6" containerName="maas-api" Oct 30 06:15:17.530289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.530193 6999 cpu_manager.go:414] "containerMap: removing container" podUID="6815f024-a76f-45e1-b16d-c1d3ed0921b3" containerName="payload-processing" Oct 30 06:15:17.530289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.530201 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6815f024-a76f-45e1-b16d-c1d3ed0921b3" containerName="payload-processing" Oct 30 06:15:17.530412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.530376 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f91b7e0-1276-4142-8b6a-eca9a5cdb1d6" containerName="maas-api" Oct 30 06:15:17.530412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.530392 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="6815f024-a76f-45e1-b16d-c1d3ed0921b3" containerName="payload-processing" Oct 30 06:15:17.530412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.530404 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="98dbc792-be6b-4283-a38b-31c945b7a537" containerName="cleanup" Oct 30 06:15:17.530501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.530417 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="98dbc792-be6b-4283-a38b-31c945b7a537" containerName="cleanup" Oct 30 06:15:17.530501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.530428 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="98dbc792-be6b-4283-a38b-31c945b7a537" containerName="cleanup" Oct 30 06:15:17.534919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.534898 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" Oct 30 06:15:17.568748 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.568711 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw"] Oct 30 06:15:17.574959 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.574905 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:17.591473 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.591442 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f7a606-dockercfg-jvbmd\"" Oct 30 06:15:17.602539 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.602462 6999 scope.go:122] "RemoveContainer" containerID="bf9c8f399190f26d92b80a8690db04d0f18b5a9ee87d5fe413ec1ac0e6154409" Oct 30 06:15:17.633786 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.632317 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f7a606\"" Oct 30 06:15:17.673531 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.656732 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4j56\" (UniqueName: \"kubernetes.io/projected/2342e52a-f07a-4069-a24b-e476995a0fbd-kube-api-access-c4j56\") pod \"payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw\" (UID: \"2342e52a-f07a-4069-a24b-e476995a0fbd\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" Oct 30 06:15:17.673531 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.659455 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2342e52a-f07a-4069-a24b-e476995a0fbd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw\" (UID: \"2342e52a-f07a-4069-a24b-e476995a0fbd\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" Oct 30 06:15:17.673531 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.670523 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" event={"ID":"98dbc792-be6b-4283-a38b-31c945b7a537","Type":"ContainerDied","Data":"86670d990d557fd5e0d5bf219f9c9770bb95f397c7eeb39babf2a365280b8827"} Oct 30 06:15:17.673531 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.670580 6999 scope.go:122] "RemoveContainer" containerID="4ecb8004c44357043f201da629889f9a18d61f83c8262ad947b691e5b4cee50c" Oct 30 06:15:17.673531 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.670688 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67" Oct 30 06:15:17.682886 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.682816 6999 generic.go:356] "Generic (PLEG): container finished" podID="57b76f03-12c8-4308-b2c7-f5261158504d" containerID="590890a1d471fd8a5a71a9be8490dbacc72034a5a2e63191a6b3f77c37155cf7" exitCode=60 Oct 30 06:15:17.683058 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.682914 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" event={"ID":"57b76f03-12c8-4308-b2c7-f5261158504d","Type":"ContainerDied","Data":"590890a1d471fd8a5a71a9be8490dbacc72034a5a2e63191a6b3f77c37155cf7"} Oct 30 06:15:17.683371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.683351 6999 scope.go:122] "RemoveContainer" containerID="590890a1d471fd8a5a71a9be8490dbacc72034a5a2e63191a6b3f77c37155cf7" Oct 30 06:15:17.684079 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:17.683695 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj_odh-ai-gateway-infra(57b76f03-12c8-4308-b2c7-f5261158504d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" podUID="57b76f03-12c8-4308-b2c7-f5261158504d" Oct 30 06:15:17.699225 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.699194 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj"] Oct 30 06:15:17.700406 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.700291 6999 scope.go:122] "RemoveContainer" containerID="0b981a79be6ff2c8e92865eb3bfbb072ed3cdfce8820107cdda0c584487eb254" Oct 30 06:15:17.762934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.761709 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2342e52a-f07a-4069-a24b-e476995a0fbd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw\" (UID: \"2342e52a-f07a-4069-a24b-e476995a0fbd\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" Oct 30 06:15:17.762934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.762056 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4j56\" (UniqueName: \"kubernetes.io/projected/2342e52a-f07a-4069-a24b-e476995a0fbd-kube-api-access-c4j56\") pod \"payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw\" (UID: \"2342e52a-f07a-4069-a24b-e476995a0fbd\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" Oct 30 06:15:17.762934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.762899 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2342e52a-f07a-4069-a24b-e476995a0fbd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw\" (UID: \"2342e52a-f07a-4069-a24b-e476995a0fbd\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" Oct 30 06:15:17.802445 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.802326 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:17.815558 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.815445 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67"] Oct 30 06:15:17.822554 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.822522 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a519e9-29889015-rrc67"] Oct 30 06:15:17.837255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.837177 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4j56\" (UniqueName: \"kubernetes.io/projected/2342e52a-f07a-4069-a24b-e476995a0fbd-kube-api-access-c4j56\") pod \"payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw\" (UID: \"2342e52a-f07a-4069-a24b-e476995a0fbd\") " pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" Oct 30 06:15:17.851744 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:17.851718 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" Oct 30 06:15:18.182502 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:18.182478 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw"] Oct 30 06:15:18.184836 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:15:18.184805 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2342e52a_f07a_4069_a24b_e476995a0fbd.slice/crio-7de06642fde544dccf58dee275be77043eba17dfb360bb5eb8eb07c4874aa3b3 WatchSource:0}: Error finding container 7de06642fde544dccf58dee275be77043eba17dfb360bb5eb8eb07c4874aa3b3: Status 404 returned error can't find the container with id 7de06642fde544dccf58dee275be77043eba17dfb360bb5eb8eb07c4874aa3b3 Oct 30 06:15:18.692384 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:18.692361 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" event={"ID":"2342e52a-f07a-4069-a24b-e476995a0fbd","Type":"ContainerStarted","Data":"de19bed6965ddb94f105e00c74f43c4bfb4e043d96733445884526564aa2dae2"} Oct 30 06:15:18.692759 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:18.692413 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" event={"ID":"2342e52a-f07a-4069-a24b-e476995a0fbd","Type":"ContainerStarted","Data":"7de06642fde544dccf58dee275be77043eba17dfb360bb5eb8eb07c4874aa3b3"} Oct 30 06:15:18.692759 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:18.692552 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" Oct 30 06:15:18.695430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:18.695407 6999 generic.go:356] "Generic (PLEG): container finished" podID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" containerID="464210d69baa3ede7175aecdff82c1d36283fe7d33b0378a0c7238a62f5c11b6" exitCode=60 Oct 30 06:15:18.695562 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:18.695484 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" event={"ID":"0a11f681-c9a2-4d78-b6ac-857b4e7d8337","Type":"ContainerDied","Data":"464210d69baa3ede7175aecdff82c1d36283fe7d33b0378a0c7238a62f5c11b6"} Oct 30 06:15:18.695562 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:18.695539 6999 scope.go:122] "RemoveContainer" containerID="bf9c8f399190f26d92b80a8690db04d0f18b5a9ee87d5fe413ec1ac0e6154409" Oct 30 06:15:18.695891 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:18.695874 6999 scope.go:122] "RemoveContainer" containerID="464210d69baa3ede7175aecdff82c1d36283fe7d33b0378a0c7238a62f5c11b6" Oct 30 06:15:18.696171 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:18.696150 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4_odh-ai-gateway-infra(0a11f681-c9a2-4d78-b6ac-857b4e7d8337)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" podUID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" Oct 30 06:15:18.738270 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:18.738191 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" podStartSLOduration=1.7381703979999998 podStartE2EDuration="1.738170398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:18.734146843 +0000 UTC m=+1285.243166727" watchObservedRunningTime="2026-10-30 06:15:18.738170398 +0000 UTC m=+1285.247190287" Oct 30 06:15:18.847754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:18.847727 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" Oct 30 06:15:18.988561 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:18.988458 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57b76f03-12c8-4308-b2c7-f5261158504d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b76f03-12c8-4308-b2c7-f5261158504d-maas-api-service-ca\") pod \"57b76f03-12c8-4308-b2c7-f5261158504d\" (UID: \"57b76f03-12c8-4308-b2c7-f5261158504d\") " Oct 30 06:15:18.988761 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:18.988740 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57b76f03-12c8-4308-b2c7-f5261158504d-kube-api-access-tkwsg\" (UniqueName: \"kubernetes.io/projected/57b76f03-12c8-4308-b2c7-f5261158504d-kube-api-access-tkwsg\") pod \"57b76f03-12c8-4308-b2c7-f5261158504d\" (UID: \"57b76f03-12c8-4308-b2c7-f5261158504d\") " Oct 30 06:15:18.989129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:18.989106 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57b76f03-12c8-4308-b2c7-f5261158504d-maas-api-service-ca" pod "57b76f03-12c8-4308-b2c7-f5261158504d" (UID: "57b76f03-12c8-4308-b2c7-f5261158504d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:18.989412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:18.989391 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57b76f03-12c8-4308-b2c7-f5261158504d-maas-api-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:18.991481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:18.991452 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57b76f03-12c8-4308-b2c7-f5261158504d-kube-api-access-tkwsg" pod "57b76f03-12c8-4308-b2c7-f5261158504d" (UID: "57b76f03-12c8-4308-b2c7-f5261158504d"). InnerVolumeSpecName "kube-api-access-tkwsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:19.091294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:19.091269 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkwsg\" (UniqueName: \"kubernetes.io/projected/57b76f03-12c8-4308-b2c7-f5261158504d-kube-api-access-tkwsg\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:19.609164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:19.609133 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98dbc792-be6b-4283-a38b-31c945b7a537" path="/var/lib/kubelet/pods/98dbc792-be6b-4283-a38b-31c945b7a537/volumes" Oct 30 06:15:19.730274 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:19.730196 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" Oct 30 06:15:19.730793 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:19.730234 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj" event={"ID":"57b76f03-12c8-4308-b2c7-f5261158504d","Type":"ContainerDied","Data":"9d344b9607472a03777213a8471888978a14ff135842cd8fdd326d2f3d4f311e"} Oct 30 06:15:19.730793 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:19.730438 6999 scope.go:122] "RemoveContainer" containerID="590890a1d471fd8a5a71a9be8490dbacc72034a5a2e63191a6b3f77c37155cf7" Oct 30 06:15:19.792884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:19.792853 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4"] Oct 30 06:15:19.800876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:19.800814 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj"] Oct 30 06:15:19.805607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:19.805578 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fe108e-29889015-qsrgj"] Oct 30 06:15:19.890847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:19.890773 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:19.947637 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:19.947614 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" Oct 30 06:15:20.105392 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:20.105356 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a11f681-c9a2-4d78-b6ac-857b4e7d8337-kube-api-access-k4rn6\" (UniqueName: \"kubernetes.io/projected/0a11f681-c9a2-4d78-b6ac-857b4e7d8337-kube-api-access-k4rn6\") pod \"0a11f681-c9a2-4d78-b6ac-857b4e7d8337\" (UID: \"0a11f681-c9a2-4d78-b6ac-857b4e7d8337\") " Oct 30 06:15:20.105577 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:20.105550 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a11f681-c9a2-4d78-b6ac-857b4e7d8337-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a11f681-c9a2-4d78-b6ac-857b4e7d8337-maas-api-service-ca\") pod \"0a11f681-c9a2-4d78-b6ac-857b4e7d8337\" (UID: \"0a11f681-c9a2-4d78-b6ac-857b4e7d8337\") " Oct 30 06:15:20.105862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:20.105842 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a11f681-c9a2-4d78-b6ac-857b4e7d8337-maas-api-service-ca" pod "0a11f681-c9a2-4d78-b6ac-857b4e7d8337" (UID: "0a11f681-c9a2-4d78-b6ac-857b4e7d8337"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:20.106150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:20.106135 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a11f681-c9a2-4d78-b6ac-857b4e7d8337-maas-api-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.108169 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:20.108140 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a11f681-c9a2-4d78-b6ac-857b4e7d8337-kube-api-access-k4rn6" pod "0a11f681-c9a2-4d78-b6ac-857b4e7d8337" (UID: "0a11f681-c9a2-4d78-b6ac-857b4e7d8337"). InnerVolumeSpecName "kube-api-access-k4rn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:20.207960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:20.207894 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4rn6\" (UniqueName: \"kubernetes.io/projected/0a11f681-c9a2-4d78-b6ac-857b4e7d8337-kube-api-access-k4rn6\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.739995 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:20.739965 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" Oct 30 06:15:20.740486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:20.739996 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4" event={"ID":"0a11f681-c9a2-4d78-b6ac-857b4e7d8337","Type":"ContainerDied","Data":"9737174155bce9b42449045aa53566603e6d13fb4ba344e6a7915f0bebbdf2ef"} Oct 30 06:15:20.740486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:20.740063 6999 scope.go:122] "RemoveContainer" containerID="464210d69baa3ede7175aecdff82c1d36283fe7d33b0378a0c7238a62f5c11b6" Oct 30 06:15:20.794918 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:20.794894 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4"] Oct 30 06:15:20.806893 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:20.806862 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-eaee88-29889015-b9vh4"] Oct 30 06:15:21.500863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:21.500832 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:21.612700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:21.612669 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" path="/var/lib/kubelet/pods/0a11f681-c9a2-4d78-b6ac-857b4e7d8337/volumes" Oct 30 06:15:21.613154 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:21.613138 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57b76f03-12c8-4308-b2c7-f5261158504d" path="/var/lib/kubelet/pods/57b76f03-12c8-4308-b2c7-f5261158504d/volumes" Oct 30 06:15:21.647585 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:21.647555 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:21.899037 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:21.899008 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:22.047981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:22.047951 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:23.759866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:23.759838 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:25.558026 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:25.557993 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:27.352900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.352868 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:27.890557 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.890520 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7318-22964b"] Oct 30 06:15:27.891194 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.891168 6999 cpu_manager.go:414] "containerMap: removing container" podUID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" containerName="cleanup" Oct 30 06:15:27.891334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.891196 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" containerName="cleanup" Oct 30 06:15:27.891334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.891244 6999 cpu_manager.go:414] "containerMap: removing container" podUID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" containerName="cleanup" Oct 30 06:15:27.891334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.891252 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" containerName="cleanup" Oct 30 06:15:27.891334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.891261 6999 cpu_manager.go:414] "containerMap: removing container" podUID="57b76f03-12c8-4308-b2c7-f5261158504d" containerName="cleanup" Oct 30 06:15:27.891334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.891269 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b76f03-12c8-4308-b2c7-f5261158504d" containerName="cleanup" Oct 30 06:15:27.891581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.891339 6999 cpu_manager.go:414] "containerMap: removing container" podUID="57b76f03-12c8-4308-b2c7-f5261158504d" containerName="cleanup" Oct 30 06:15:27.891581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.891347 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b76f03-12c8-4308-b2c7-f5261158504d" containerName="cleanup" Oct 30 06:15:27.891581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.891359 6999 cpu_manager.go:414] "containerMap: removing container" podUID="98dbc792-be6b-4283-a38b-31c945b7a537" containerName="cleanup" Oct 30 06:15:27.891581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.891366 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98dbc792-be6b-4283-a38b-31c945b7a537" containerName="cleanup" Oct 30 06:15:27.891581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.891526 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" containerName="cleanup" Oct 30 06:15:27.891581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.891552 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b76f03-12c8-4308-b2c7-f5261158504d" containerName="cleanup" Oct 30 06:15:27.891581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.891564 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b76f03-12c8-4308-b2c7-f5261158504d" containerName="cleanup" Oct 30 06:15:27.891581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.891575 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" containerName="cleanup" Oct 30 06:15:27.891952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.891586 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="57b76f03-12c8-4308-b2c7-f5261158504d" containerName="cleanup" Oct 30 06:15:27.896574 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.896552 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7318-22964b" Oct 30 06:15:27.910802 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.910775 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7318-22964b"] Oct 30 06:15:27.919446 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:27.919412 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p6zmk\"" Oct 30 06:15:28.031061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:28.031029 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shmqn\" (UniqueName: \"kubernetes.io/projected/384437e4-e890-457a-90fc-f41ddbce25e6-kube-api-access-shmqn\") pod \"test-curl-7318-22964b\" (UID: \"384437e4-e890-457a-90fc-f41ddbce25e6\") " pod="opendatahub/test-curl-7318-22964b" Oct 30 06:15:28.132853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:28.132823 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shmqn\" (UniqueName: \"kubernetes.io/projected/384437e4-e890-457a-90fc-f41ddbce25e6-kube-api-access-shmqn\") pod \"test-curl-7318-22964b\" (UID: \"384437e4-e890-457a-90fc-f41ddbce25e6\") " pod="opendatahub/test-curl-7318-22964b" Oct 30 06:15:28.157766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:28.157695 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shmqn\" (UniqueName: \"kubernetes.io/projected/384437e4-e890-457a-90fc-f41ddbce25e6-kube-api-access-shmqn\") pod \"test-curl-7318-22964b\" (UID: \"384437e4-e890-457a-90fc-f41ddbce25e6\") " pod="opendatahub/test-curl-7318-22964b" Oct 30 06:15:28.209008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:28.208976 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7318-22964b" Oct 30 06:15:28.516087 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:28.516058 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7318-22964b"] Oct 30 06:15:28.517259 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:15:28.517235 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod384437e4_e890_457a_90fc_f41ddbce25e6.slice/crio-50e335b3d612c4ba06c2a479382359fb7b8783ec8be5129b06433d06d4b56f9e WatchSource:0}: Error finding container 50e335b3d612c4ba06c2a479382359fb7b8783ec8be5129b06433d06d4b56f9e: Status 404 returned error can't find the container with id 50e335b3d612c4ba06c2a479382359fb7b8783ec8be5129b06433d06d4b56f9e Oct 30 06:15:28.798057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:28.797978 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7318-22964b" event={"ID":"384437e4-e890-457a-90fc-f41ddbce25e6","Type":"ContainerStarted","Data":"381aecc145b08972b6dfa23f1869fe8dc300502f242921051e35d9b17376fb29"} Oct 30 06:15:28.798057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:28.798016 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7318-22964b" event={"ID":"384437e4-e890-457a-90fc-f41ddbce25e6","Type":"ContainerStarted","Data":"50e335b3d612c4ba06c2a479382359fb7b8783ec8be5129b06433d06d4b56f9e"} Oct 30 06:15:28.836982 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:28.836937 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7318-22964b" podStartSLOduration=1.8369223 podStartE2EDuration="1.8369223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:28.835679558 +0000 UTC m=+1295.344699442" watchObservedRunningTime="2026-10-30 06:15:28.8369223 +0000 UTC m=+1295.345942188" Oct 30 06:15:29.246818 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:29.246785 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:29.735347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:29.735321 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" Oct 30 06:15:29.754069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:29.754043 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7318-22964b"] Oct 30 06:15:29.766205 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:29.766182 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7318-22964b"] Oct 30 06:15:30.811799 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:30.811763 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7318-22964b" podUID="384437e4-e890-457a-90fc-f41ddbce25e6" containerName="test-curl-7318-22964b" containerID="cri-o://381aecc145b08972b6dfa23f1869fe8dc300502f242921051e35d9b17376fb29" gracePeriod=2 Oct 30 06:15:30.825427 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:30.825387 6999 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7318-22964b\" is forbidden: User \"system:node:ip-10-0-131-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-226.ec2.internal' and this object" podUID="384437e4-e890-457a-90fc-f41ddbce25e6" pod="opendatahub/test-curl-7318-22964b" Oct 30 06:15:31.228056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:31.228024 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:32.968347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:32.968317 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8"] Oct 30 06:15:32.968957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:32.968940 6999 cpu_manager.go:414] "containerMap: removing container" podUID="57b76f03-12c8-4308-b2c7-f5261158504d" containerName="cleanup" Oct 30 06:15:32.968957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:32.968956 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57b76f03-12c8-4308-b2c7-f5261158504d" containerName="cleanup" Oct 30 06:15:32.969119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:32.968966 6999 cpu_manager.go:414] "containerMap: removing container" podUID="384437e4-e890-457a-90fc-f41ddbce25e6" containerName="test-curl-7318-22964b" Oct 30 06:15:32.969119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:32.968972 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="384437e4-e890-457a-90fc-f41ddbce25e6" containerName="test-curl-7318-22964b" Oct 30 06:15:32.969119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:32.969026 6999 cpu_manager.go:414] "containerMap: removing container" podUID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" containerName="cleanup" Oct 30 06:15:32.969119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:32.969032 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" containerName="cleanup" Oct 30 06:15:32.969388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:32.969142 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a11f681-c9a2-4d78-b6ac-857b4e7d8337" containerName="cleanup" Oct 30 06:15:32.969388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:32.969165 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="384437e4-e890-457a-90fc-f41ddbce25e6" containerName="test-curl-7318-22964b" Oct 30 06:15:32.973002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:32.972981 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:32.988389 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:32.988364 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f7a606-serving-cert\"" Oct 30 06:15:33.006815 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.005308 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8"] Oct 30 06:15:33.055581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.055554 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7318-22964b" Oct 30 06:15:33.095953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.095892 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:33.106542 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.106507 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4628f823-8667-4403-8fd5-2c27c8c76c23-openshift-service-ca\") pod \"maas-api-e2e-api-b-f7a606-59df8df945-qskw8\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:33.106664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.106552 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-maas-api-tls\") pod \"maas-api-e2e-api-b-f7a606-59df8df945-qskw8\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:33.106711 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.106666 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-metrics-certs\") pod \"maas-api-e2e-api-b-f7a606-59df8df945-qskw8\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:33.106752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.106712 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfzvf\" (UniqueName: \"kubernetes.io/projected/4628f823-8667-4403-8fd5-2c27c8c76c23-kube-api-access-gfzvf\") pod \"maas-api-e2e-api-b-f7a606-59df8df945-qskw8\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:33.208309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.208262 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/384437e4-e890-457a-90fc-f41ddbce25e6-kube-api-access-shmqn\" (UniqueName: \"kubernetes.io/projected/384437e4-e890-457a-90fc-f41ddbce25e6-kube-api-access-shmqn\") pod \"384437e4-e890-457a-90fc-f41ddbce25e6\" (UID: \"384437e4-e890-457a-90fc-f41ddbce25e6\") " Oct 30 06:15:33.208585 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.208566 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfzvf\" (UniqueName: \"kubernetes.io/projected/4628f823-8667-4403-8fd5-2c27c8c76c23-kube-api-access-gfzvf\") pod \"maas-api-e2e-api-b-f7a606-59df8df945-qskw8\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:33.208822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.208808 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4628f823-8667-4403-8fd5-2c27c8c76c23-openshift-service-ca\") pod \"maas-api-e2e-api-b-f7a606-59df8df945-qskw8\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:33.209435 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.208842 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-maas-api-tls\") pod \"maas-api-e2e-api-b-f7a606-59df8df945-qskw8\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:33.209435 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.208922 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-metrics-certs\") pod \"maas-api-e2e-api-b-f7a606-59df8df945-qskw8\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:33.209435 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:33.209207 6999 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-serving-cert: secret "maas-api-e2e-api-b-f7a606-serving-cert" not found Oct 30 06:15:33.209435 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:33.209321 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-maas-api-tls podName:4628f823-8667-4403-8fd5-2c27c8c76c23 nodeName:}" failed. No retries permitted until 2026-10-30 06:15:33.709299088 +0000 UTC m=+1300.218318967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-maas-api-tls") pod "maas-api-e2e-api-b-f7a606-59df8df945-qskw8" (UID: "4628f823-8667-4403-8fd5-2c27c8c76c23") : secret "maas-api-e2e-api-b-f7a606-serving-cert" not found Oct 30 06:15:33.209688 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.209612 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4628f823-8667-4403-8fd5-2c27c8c76c23-openshift-service-ca\") pod \"maas-api-e2e-api-b-f7a606-59df8df945-qskw8\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:33.210928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.210902 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/384437e4-e890-457a-90fc-f41ddbce25e6-kube-api-access-shmqn" pod "384437e4-e890-457a-90fc-f41ddbce25e6" (UID: "384437e4-e890-457a-90fc-f41ddbce25e6"). InnerVolumeSpecName "kube-api-access-shmqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:33.212005 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.211948 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-metrics-certs\") pod \"maas-api-e2e-api-b-f7a606-59df8df945-qskw8\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:33.235976 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.235942 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfzvf\" (UniqueName: \"kubernetes.io/projected/4628f823-8667-4403-8fd5-2c27c8c76c23-kube-api-access-gfzvf\") pod \"maas-api-e2e-api-b-f7a606-59df8df945-qskw8\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:33.249056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.249024 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:33.310088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.310058 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shmqn\" (UniqueName: \"kubernetes.io/projected/384437e4-e890-457a-90fc-f41ddbce25e6-kube-api-access-shmqn\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:33.395858 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.395828 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:33.609261 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.609229 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="384437e4-e890-457a-90fc-f41ddbce25e6" path="/var/lib/kubelet/pods/384437e4-e890-457a-90fc-f41ddbce25e6/volumes" Oct 30 06:15:33.715962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.715885 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-maas-api-tls\") pod \"maas-api-e2e-api-b-f7a606-59df8df945-qskw8\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:33.718419 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.718399 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-maas-api-tls\") pod \"maas-api-e2e-api-b-f7a606-59df8df945-qskw8\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:33.834715 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.834677 6999 generic.go:356] "Generic (PLEG): container finished" podID="384437e4-e890-457a-90fc-f41ddbce25e6" containerID="381aecc145b08972b6dfa23f1869fe8dc300502f242921051e35d9b17376fb29" exitCode=137 Oct 30 06:15:33.834901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.834729 6999 scope.go:122] "RemoveContainer" containerID="381aecc145b08972b6dfa23f1869fe8dc300502f242921051e35d9b17376fb29" Oct 30 06:15:33.834901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.834729 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7318-22964b" Oct 30 06:15:33.850513 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.850311 6999 scope.go:122] "RemoveContainer" containerID="381aecc145b08972b6dfa23f1869fe8dc300502f242921051e35d9b17376fb29" Oct 30 06:15:33.850695 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:33.850647 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"381aecc145b08972b6dfa23f1869fe8dc300502f242921051e35d9b17376fb29\": container with ID starting with 381aecc145b08972b6dfa23f1869fe8dc300502f242921051e35d9b17376fb29 not found: ID does not exist" containerID="381aecc145b08972b6dfa23f1869fe8dc300502f242921051e35d9b17376fb29" Oct 30 06:15:33.850695 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.850680 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"381aecc145b08972b6dfa23f1869fe8dc300502f242921051e35d9b17376fb29"} err="failed to get container status \"381aecc145b08972b6dfa23f1869fe8dc300502f242921051e35d9b17376fb29\": rpc error: code = NotFound desc = could not find container \"381aecc145b08972b6dfa23f1869fe8dc300502f242921051e35d9b17376fb29\": container with ID starting with 381aecc145b08972b6dfa23f1869fe8dc300502f242921051e35d9b17376fb29 not found: ID does not exist" Oct 30 06:15:33.887796 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:33.887770 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:34.143822 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:34.143784 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:15:34.272319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:34.272284 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8"] Oct 30 06:15:34.845519 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:34.845486 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" event={"ID":"4628f823-8667-4403-8fd5-2c27c8c76c23","Type":"ContainerStarted","Data":"10e1c137acdca01b43ed9a6593113224649f31e04c2c06676862672c5679b2d3"} Oct 30 06:15:34.845686 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:34.845535 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" event={"ID":"4628f823-8667-4403-8fd5-2c27c8c76c23","Type":"ContainerStarted","Data":"e6bea4bde7fd491e26675cb46acbc39aa413dedd07c44e126df872f70b84289c"} Oct 30 06:15:34.845686 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:34.845622 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:34.891395 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:34.891301 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" podStartSLOduration=2.529372312 podStartE2EDuration="2.891288165s" totalImagesPullingTime="361.915853ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:34.888059648 +0000 UTC m=+1301.397079561" watchObservedRunningTime="2026-10-30 06:15:34.891288165 +0000 UTC m=+1301.400308051" Oct 30 06:15:35.039068 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:35.039029 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:37.052438 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:37.052376 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:38.057188 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.057162 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:38.518512 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.518478 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj"] Oct 30 06:15:38.518767 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.518739 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" podUID="1bcbb1eb-21c8-4b95-a209-8d94d54f719d" containerName="maas-api" containerID="cri-o://c9cd77fd96f1dc7794f933349cec8d044751e1070a2f449526e678f9dd236fc9" gracePeriod=30 Oct 30 06:15:38.595889 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:38.595854 6999 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bcbb1eb_21c8_4b95_a209_8d94d54f719d.slice/crio-c9cd77fd96f1dc7794f933349cec8d044751e1070a2f449526e678f9dd236fc9.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:15:38.622508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.622463 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:38.883761 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.883727 6999 generic.go:356] "Generic (PLEG): container finished" podID="1bcbb1eb-21c8-4b95-a209-8d94d54f719d" containerID="c9cd77fd96f1dc7794f933349cec8d044751e1070a2f449526e678f9dd236fc9" exitCode=0 Oct 30 06:15:38.883952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.883936 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" event={"ID":"1bcbb1eb-21c8-4b95-a209-8d94d54f719d","Type":"ContainerDied","Data":"c9cd77fd96f1dc7794f933349cec8d044751e1070a2f449526e678f9dd236fc9"} Oct 30 06:15:38.884002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.883987 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" event={"ID":"1bcbb1eb-21c8-4b95-a209-8d94d54f719d","Type":"ContainerDied","Data":"14a211b5d32f654103739a14d4f03fe664367119e25371cfc99d4fed98cb6cf2"} Oct 30 06:15:38.884037 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.884018 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14a211b5d32f654103739a14d4f03fe664367119e25371cfc99d4fed98cb6cf2" Oct 30 06:15:38.896443 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.896408 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:15:38.924794 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.924743 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-maas-api-tls\") pod \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " Oct 30 06:15:38.924991 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.924840 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-openshift-service-ca\") pod \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " Oct 30 06:15:38.924991 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.924973 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-metrics-certs\") pod \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " Oct 30 06:15:38.925116 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.925002 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-kube-api-access-lxktz\" (UniqueName: \"kubernetes.io/projected/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-kube-api-access-lxktz\") pod \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\" (UID: \"1bcbb1eb-21c8-4b95-a209-8d94d54f719d\") " Oct 30 06:15:38.925651 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.925604 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-openshift-service-ca" pod "1bcbb1eb-21c8-4b95-a209-8d94d54f719d" (UID: "1bcbb1eb-21c8-4b95-a209-8d94d54f719d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:38.927918 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.927860 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-metrics-certs" pod "1bcbb1eb-21c8-4b95-a209-8d94d54f719d" (UID: "1bcbb1eb-21c8-4b95-a209-8d94d54f719d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:15:38.928044 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.927969 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-kube-api-access-lxktz" pod "1bcbb1eb-21c8-4b95-a209-8d94d54f719d" (UID: "1bcbb1eb-21c8-4b95-a209-8d94d54f719d"). InnerVolumeSpecName "kube-api-access-lxktz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:38.928349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:38.928321 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-maas-api-tls" pod "1bcbb1eb-21c8-4b95-a209-8d94d54f719d" (UID: "1bcbb1eb-21c8-4b95-a209-8d94d54f719d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:15:39.026847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:39.026818 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:39.026847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:39.026845 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:39.026847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:39.026855 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxktz\" (UniqueName: \"kubernetes.io/projected/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-kube-api-access-lxktz\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:39.027073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:39.026864 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bcbb1eb-21c8-4b95-a209-8d94d54f719d-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:39.183061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:39.183021 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:39.340376 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:39.340339 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:39.495052 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:39.493125 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:39.886751 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:39.886720 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:39.890322 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:39.890302 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj" Oct 30 06:15:39.966068 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:39.966037 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj"] Oct 30 06:15:39.978274 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:39.978248 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e6df62-8559497db9-7vtvj"] Oct 30 06:15:40.767770 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:40.767729 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:40.859947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:40.859918 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:15:41.083429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:41.083351 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:41.610457 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:41.610425 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bcbb1eb-21c8-4b95-a209-8d94d54f719d" path="/var/lib/kubelet/pods/1bcbb1eb-21c8-4b95-a209-8d94d54f719d/volumes" Oct 30 06:15:41.712291 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:41.712261 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:43.543067 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:43.543031 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:44.654448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:44.654412 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:45.395166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:45.395127 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:46.583533 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:46.583500 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:46.730790 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:46.730754 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:46.888834 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:46.888795 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d"] Oct 30 06:15:46.889106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:46.889073 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" podUID="c4989e46-f4b8-4113-aea1-66994ea9c6c6" containerName="istio-proxy" containerID="cri-o://2c6553135b536ef6c7622ca4109a42165dccc521cfe3afbb0507826cd9bf17b9" gracePeriod=30 Oct 30 06:15:46.958092 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:46.958062 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:47.193758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:47.193676 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:49.040111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:49.040080 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:50.939083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:50.939047 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:52.170360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.170305 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:15:52.317115 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.317078 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-credential-socket\") pod \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " Oct 30 06:15:52.317341 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.317128 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-workload-socket\") pod \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " Oct 30 06:15:52.317341 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.317309 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-token\" (UniqueName: \"kubernetes.io/projected/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-token\") pod \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " Oct 30 06:15:52.317466 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.317350 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-data\") pod \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " Oct 30 06:15:52.317466 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.317384 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-credential-socket" pod "c4989e46-f4b8-4113-aea1-66994ea9c6c6" (UID: "c4989e46-f4b8-4113-aea1-66994ea9c6c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:52.317466 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.317432 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4989e46-f4b8-4113-aea1-66994ea9c6c6-kube-api-access-q9cgw\" (UniqueName: \"kubernetes.io/projected/c4989e46-f4b8-4113-aea1-66994ea9c6c6-kube-api-access-q9cgw\") pod \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " Oct 30 06:15:52.317617 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.317472 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-workload-certs\") pod \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " Oct 30 06:15:52.317617 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.317484 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-workload-socket" pod "c4989e46-f4b8-4113-aea1-66994ea9c6c6" (UID: "c4989e46-f4b8-4113-aea1-66994ea9c6c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:52.317617 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.317560 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-envoy\") pod \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " Oct 30 06:15:52.317765 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.317645 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istiod-ca-cert\") pod \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " Oct 30 06:15:52.317765 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.317707 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-data" pod "c4989e46-f4b8-4113-aea1-66994ea9c6c6" (UID: "c4989e46-f4b8-4113-aea1-66994ea9c6c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:52.317765 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.317735 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-workload-certs" pod "c4989e46-f4b8-4113-aea1-66994ea9c6c6" (UID: "c4989e46-f4b8-4113-aea1-66994ea9c6c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:52.317765 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.317728 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-podinfo\") pod \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\" (UID: \"c4989e46-f4b8-4113-aea1-66994ea9c6c6\") " Oct 30 06:15:52.318062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.318037 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istiod-ca-cert" pod "c4989e46-f4b8-4113-aea1-66994ea9c6c6" (UID: "c4989e46-f4b8-4113-aea1-66994ea9c6c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:52.318421 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.318396 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.318509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.318425 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.318509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.318439 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.318509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.318451 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.318509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.318460 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.320361 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.320331 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-envoy" pod "c4989e46-f4b8-4113-aea1-66994ea9c6c6" (UID: "c4989e46-f4b8-4113-aea1-66994ea9c6c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:52.320452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.320355 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-podinfo" pod "c4989e46-f4b8-4113-aea1-66994ea9c6c6" (UID: "c4989e46-f4b8-4113-aea1-66994ea9c6c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:15:52.320452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.320410 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-token" pod "c4989e46-f4b8-4113-aea1-66994ea9c6c6" (UID: "c4989e46-f4b8-4113-aea1-66994ea9c6c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:52.320452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.320438 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4989e46-f4b8-4113-aea1-66994ea9c6c6-kube-api-access-q9cgw" pod "c4989e46-f4b8-4113-aea1-66994ea9c6c6" (UID: "c4989e46-f4b8-4113-aea1-66994ea9c6c6"). InnerVolumeSpecName "kube-api-access-q9cgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:52.419961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.419939 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.419961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.419963 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9cgw\" (UniqueName: \"kubernetes.io/projected/c4989e46-f4b8-4113-aea1-66994ea9c6c6-kube-api-access-q9cgw\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.420097 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.419975 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.420097 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.419984 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4989e46-f4b8-4113-aea1-66994ea9c6c6-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:52.823997 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.823961 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:52.990570 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.990536 6999 generic.go:356] "Generic (PLEG): container finished" podID="c4989e46-f4b8-4113-aea1-66994ea9c6c6" containerID="2c6553135b536ef6c7622ca4109a42165dccc521cfe3afbb0507826cd9bf17b9" exitCode=0 Oct 30 06:15:52.990761 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.990592 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" Oct 30 06:15:52.990761 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.990616 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" event={"ID":"c4989e46-f4b8-4113-aea1-66994ea9c6c6","Type":"ContainerDied","Data":"2c6553135b536ef6c7622ca4109a42165dccc521cfe3afbb0507826cd9bf17b9"} Oct 30 06:15:52.990761 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.990671 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d" event={"ID":"c4989e46-f4b8-4113-aea1-66994ea9c6c6","Type":"ContainerDied","Data":"cdadbac51865e52550f671b66fe6d9c102b8a48d6e97f7bb04d2f8c4ad45a23a"} Oct 30 06:15:52.990761 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:52.990695 6999 scope.go:122] "RemoveContainer" containerID="2c6553135b536ef6c7622ca4109a42165dccc521cfe3afbb0507826cd9bf17b9" Oct 30 06:15:53.002174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:53.002157 6999 scope.go:122] "RemoveContainer" containerID="2c6553135b536ef6c7622ca4109a42165dccc521cfe3afbb0507826cd9bf17b9" Oct 30 06:15:53.002495 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:53.002471 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6553135b536ef6c7622ca4109a42165dccc521cfe3afbb0507826cd9bf17b9\": container with ID starting with 2c6553135b536ef6c7622ca4109a42165dccc521cfe3afbb0507826cd9bf17b9 not found: ID does not exist" containerID="2c6553135b536ef6c7622ca4109a42165dccc521cfe3afbb0507826cd9bf17b9" Oct 30 06:15:53.002607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:53.002501 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6553135b536ef6c7622ca4109a42165dccc521cfe3afbb0507826cd9bf17b9"} err="failed to get container status \"2c6553135b536ef6c7622ca4109a42165dccc521cfe3afbb0507826cd9bf17b9\": rpc error: code = NotFound desc = could not find container \"2c6553135b536ef6c7622ca4109a42165dccc521cfe3afbb0507826cd9bf17b9\": container with ID starting with 2c6553135b536ef6c7622ca4109a42165dccc521cfe3afbb0507826cd9bf17b9 not found: ID does not exist" Oct 30 06:15:53.043163 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:53.043137 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d"] Oct 30 06:15:53.055088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:53.055068 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e6df62-openshift-default-7cc85f6f9f-slc8d"] Oct 30 06:15:53.609159 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:53.609132 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4989e46-f4b8-4113-aea1-66994ea9c6c6" path="/var/lib/kubelet/pods/c4989e46-f4b8-4113-aea1-66994ea9c6c6/volumes" Oct 30 06:15:53.928895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:53.922441 6999 scope.go:122] "RemoveContainer" containerID="32888dd8882c6ba67a05fc16d14821345c66a06ca184506b69e299196a1840e2" Oct 30 06:15:53.940078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:53.940059 6999 scope.go:122] "RemoveContainer" containerID="4485ee3130064931e516b8601c11574f8bf762c7b1eab9fb9c52ad59f16c9aaf" Oct 30 06:15:54.617910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:54.617874 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:55.033590 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:55.033505 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg"] Oct 30 06:15:55.045565 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:55.045535 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ca66358-jwlrg"] Oct 30 06:15:55.609500 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:55.609469 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8168c11e-ee79-4ffb-a53f-4f10563ff965" path="/var/lib/kubelet/pods/8168c11e-ee79-4ffb-a53f-4f10563ff965/volumes" Oct 30 06:15:56.038900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.038810 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:56.138590 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.138551 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff"] Oct 30 06:15:56.139477 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.139428 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" podUID="12365d0d-c44f-488e-9362-d5f74d4de585" containerName="maas-api" containerID="cri-o://6eead7fa7571004f80711472bf8549155abf2a85c0157191d222b25b9786e16e" gracePeriod=30 Oct 30 06:15:56.244208 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.244160 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:56.456754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.456713 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:15:56.480689 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.480656 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12365d0d-c44f-488e-9362-d5f74d4de585-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12365d0d-c44f-488e-9362-d5f74d4de585-openshift-service-ca\") pod \"12365d0d-c44f-488e-9362-d5f74d4de585\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " Oct 30 06:15:56.481055 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.481023 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12365d0d-c44f-488e-9362-d5f74d4de585-kube-api-access-mnsj4\" (UniqueName: \"kubernetes.io/projected/12365d0d-c44f-488e-9362-d5f74d4de585-kube-api-access-mnsj4\") pod \"12365d0d-c44f-488e-9362-d5f74d4de585\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " Oct 30 06:15:56.481328 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.481311 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12365d0d-c44f-488e-9362-d5f74d4de585-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12365d0d-c44f-488e-9362-d5f74d4de585-maas-api-tls\") pod \"12365d0d-c44f-488e-9362-d5f74d4de585\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " Oct 30 06:15:56.481485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.481470 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12365d0d-c44f-488e-9362-d5f74d4de585-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12365d0d-c44f-488e-9362-d5f74d4de585-metrics-certs\") pod \"12365d0d-c44f-488e-9362-d5f74d4de585\" (UID: \"12365d0d-c44f-488e-9362-d5f74d4de585\") " Oct 30 06:15:56.489437 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.489385 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12365d0d-c44f-488e-9362-d5f74d4de585-openshift-service-ca" pod "12365d0d-c44f-488e-9362-d5f74d4de585" (UID: "12365d0d-c44f-488e-9362-d5f74d4de585"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:56.493044 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.493017 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12365d0d-c44f-488e-9362-d5f74d4de585-kube-api-access-mnsj4" pod "12365d0d-c44f-488e-9362-d5f74d4de585" (UID: "12365d0d-c44f-488e-9362-d5f74d4de585"). InnerVolumeSpecName "kube-api-access-mnsj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:56.495058 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.494569 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12365d0d-c44f-488e-9362-d5f74d4de585-metrics-certs" pod "12365d0d-c44f-488e-9362-d5f74d4de585" (UID: "12365d0d-c44f-488e-9362-d5f74d4de585"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:15:56.514305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.496389 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12365d0d-c44f-488e-9362-d5f74d4de585-maas-api-tls" pod "12365d0d-c44f-488e-9362-d5f74d4de585" (UID: "12365d0d-c44f-488e-9362-d5f74d4de585"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:15:56.584609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.584460 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12365d0d-c44f-488e-9362-d5f74d4de585-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:56.584609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.584498 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnsj4\" (UniqueName: \"kubernetes.io/projected/12365d0d-c44f-488e-9362-d5f74d4de585-kube-api-access-mnsj4\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:56.584609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.584514 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12365d0d-c44f-488e-9362-d5f74d4de585-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:56.584609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.584528 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12365d0d-c44f-488e-9362-d5f74d4de585-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:15:56.915708 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:56.915678 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:57.016430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:57.016399 6999 generic.go:356] "Generic (PLEG): container finished" podID="12365d0d-c44f-488e-9362-d5f74d4de585" containerID="6eead7fa7571004f80711472bf8549155abf2a85c0157191d222b25b9786e16e" exitCode=0 Oct 30 06:15:57.016593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:57.016450 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" Oct 30 06:15:57.016593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:57.016484 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" event={"ID":"12365d0d-c44f-488e-9362-d5f74d4de585","Type":"ContainerDied","Data":"6eead7fa7571004f80711472bf8549155abf2a85c0157191d222b25b9786e16e"} Oct 30 06:15:57.016593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:57.016540 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff" event={"ID":"12365d0d-c44f-488e-9362-d5f74d4de585","Type":"ContainerDied","Data":"c238288a2b95c9bc831249b7fb21b4840df512c619590f4fb6e1c733c19eb475"} Oct 30 06:15:57.016593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:57.016571 6999 scope.go:122] "RemoveContainer" containerID="6eead7fa7571004f80711472bf8549155abf2a85c0157191d222b25b9786e16e" Oct 30 06:15:57.027840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:57.027820 6999 scope.go:122] "RemoveContainer" containerID="6eead7fa7571004f80711472bf8549155abf2a85c0157191d222b25b9786e16e" Oct 30 06:15:57.028139 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:15:57.028119 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eead7fa7571004f80711472bf8549155abf2a85c0157191d222b25b9786e16e\": container with ID starting with 6eead7fa7571004f80711472bf8549155abf2a85c0157191d222b25b9786e16e not found: ID does not exist" containerID="6eead7fa7571004f80711472bf8549155abf2a85c0157191d222b25b9786e16e" Oct 30 06:15:57.028260 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:57.028148 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eead7fa7571004f80711472bf8549155abf2a85c0157191d222b25b9786e16e"} err="failed to get container status \"6eead7fa7571004f80711472bf8549155abf2a85c0157191d222b25b9786e16e\": rpc error: code = NotFound desc = could not find container \"6eead7fa7571004f80711472bf8549155abf2a85c0157191d222b25b9786e16e\": container with ID starting with 6eead7fa7571004f80711472bf8549155abf2a85c0157191d222b25b9786e16e not found: ID does not exist" Oct 30 06:15:57.073198 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:57.073165 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff"] Oct 30 06:15:57.075008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:57.074977 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:57.086545 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:57.086504 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ba44b248-56f4f88c44-qc8ff"] Oct 30 06:15:57.571352 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:57.571313 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:57.609289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:57.609262 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12365d0d-c44f-488e-9362-d5f74d4de585" path="/var/lib/kubelet/pods/12365d0d-c44f-488e-9362-d5f74d4de585/volumes" Oct 30 06:15:58.145192 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:58.145162 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:58.295873 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:58.295840 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:59.032363 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:59.032322 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:15:59.421580 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:15:59.421552 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:00.824553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:00.824521 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:01.304677 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:01.304647 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:02.649499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:02.649465 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:02.844208 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:02.844176 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:04.149845 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:16:04.149799 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:16:04.646706 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:04.646677 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:06.540998 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:06.540962 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:07.034150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:07.034115 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j"] Oct 30 06:16:07.045934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:07.045909 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-482cb1aa-mxt4j"] Oct 30 06:16:07.609047 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:07.609017 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2e4084a-aabb-49f7-bfbb-6a0d4f14e202" path="/var/lib/kubelet/pods/c2e4084a-aabb-49f7-bfbb-6a0d4f14e202/volumes" Oct 30 06:16:07.770619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:07.770588 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw"] Oct 30 06:16:07.912593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:07.912508 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:08.060054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:08.060022 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:08.454850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:08.454822 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:10.259883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:10.259851 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:12.104489 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:12.104449 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:12.771838 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:12.771799 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" podUID="ff79aff8-16c9-4edf-8b16-81ef85272ae5" containerName="payload-pre-processing" containerID="cri-o://31cf9288793ca3b5edd4ab11565401ac47ee8b3bda495dff62d764b4cd5b07f0" gracePeriod=25 Oct 30 06:16:13.051901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.051873 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" Oct 30 06:16:13.136231 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.136192 6999 generic.go:356] "Generic (PLEG): container finished" podID="ff79aff8-16c9-4edf-8b16-81ef85272ae5" containerID="31cf9288793ca3b5edd4ab11565401ac47ee8b3bda495dff62d764b4cd5b07f0" exitCode=0 Oct 30 06:16:13.136677 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.136256 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" Oct 30 06:16:13.136677 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.136264 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" event={"ID":"ff79aff8-16c9-4edf-8b16-81ef85272ae5","Type":"ContainerDied","Data":"31cf9288793ca3b5edd4ab11565401ac47ee8b3bda495dff62d764b4cd5b07f0"} Oct 30 06:16:13.136677 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.136316 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw" event={"ID":"ff79aff8-16c9-4edf-8b16-81ef85272ae5","Type":"ContainerDied","Data":"12e27c026a5ba885f073ad35f19684681925f8e2a65f9cf712bd7e231f1c5bf3"} Oct 30 06:16:13.136677 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.136339 6999 scope.go:122] "RemoveContainer" containerID="31cf9288793ca3b5edd4ab11565401ac47ee8b3bda495dff62d764b4cd5b07f0" Oct 30 06:16:13.146242 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.146206 6999 scope.go:122] "RemoveContainer" containerID="31cf9288793ca3b5edd4ab11565401ac47ee8b3bda495dff62d764b4cd5b07f0" Oct 30 06:16:13.146545 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:16:13.146526 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31cf9288793ca3b5edd4ab11565401ac47ee8b3bda495dff62d764b4cd5b07f0\": container with ID starting with 31cf9288793ca3b5edd4ab11565401ac47ee8b3bda495dff62d764b4cd5b07f0 not found: ID does not exist" containerID="31cf9288793ca3b5edd4ab11565401ac47ee8b3bda495dff62d764b4cd5b07f0" Oct 30 06:16:13.146597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.146555 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31cf9288793ca3b5edd4ab11565401ac47ee8b3bda495dff62d764b4cd5b07f0"} err="failed to get container status \"31cf9288793ca3b5edd4ab11565401ac47ee8b3bda495dff62d764b4cd5b07f0\": rpc error: code = NotFound desc = could not find container \"31cf9288793ca3b5edd4ab11565401ac47ee8b3bda495dff62d764b4cd5b07f0\": container with ID starting with 31cf9288793ca3b5edd4ab11565401ac47ee8b3bda495dff62d764b4cd5b07f0 not found: ID does not exist" Oct 30 06:16:13.217193 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.217169 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff79aff8-16c9-4edf-8b16-81ef85272ae5-kube-api-access-mwb2z\" (UniqueName: \"kubernetes.io/projected/ff79aff8-16c9-4edf-8b16-81ef85272ae5-kube-api-access-mwb2z\") pod \"ff79aff8-16c9-4edf-8b16-81ef85272ae5\" (UID: \"ff79aff8-16c9-4edf-8b16-81ef85272ae5\") " Oct 30 06:16:13.217359 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.217347 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff79aff8-16c9-4edf-8b16-81ef85272ae5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff79aff8-16c9-4edf-8b16-81ef85272ae5-plugins-config-volume\") pod \"ff79aff8-16c9-4edf-8b16-81ef85272ae5\" (UID: \"ff79aff8-16c9-4edf-8b16-81ef85272ae5\") " Oct 30 06:16:13.217678 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.217657 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff79aff8-16c9-4edf-8b16-81ef85272ae5-plugins-config-volume" pod "ff79aff8-16c9-4edf-8b16-81ef85272ae5" (UID: "ff79aff8-16c9-4edf-8b16-81ef85272ae5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:13.219510 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.219484 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff79aff8-16c9-4edf-8b16-81ef85272ae5-kube-api-access-mwb2z" pod "ff79aff8-16c9-4edf-8b16-81ef85272ae5" (UID: "ff79aff8-16c9-4edf-8b16-81ef85272ae5"). InnerVolumeSpecName "kube-api-access-mwb2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:13.318934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.318859 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwb2z\" (UniqueName: \"kubernetes.io/projected/ff79aff8-16c9-4edf-8b16-81ef85272ae5-kube-api-access-mwb2z\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:16:13.318934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.318884 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff79aff8-16c9-4edf-8b16-81ef85272ae5-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:16:13.482175 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.482144 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw"] Oct 30 06:16:13.488080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.488057 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c15d3a-7fd99699b7-fprtw"] Oct 30 06:16:13.609139 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.609102 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff79aff8-16c9-4edf-8b16-81ef85272ae5" path="/var/lib/kubelet/pods/ff79aff8-16c9-4edf-8b16-81ef85272ae5/volumes" Oct 30 06:16:13.895427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:13.895348 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:15.706825 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:15.706789 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:17.640126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:17.640094 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:19.739002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:19.738956 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:20.036516 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:20.036423 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn"] Oct 30 06:16:20.048126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:20.048096 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e150dea1-dktxn"] Oct 30 06:16:21.647237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:21.632110 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26908887-9f3f-4e52-9c93-799e9581a075" path="/var/lib/kubelet/pods/26908887-9f3f-4e52-9c93-799e9581a075/volumes" Oct 30 06:16:22.050388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:22.050289 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" podUID="5990b367-6bd6-483b-ab76-35375b015cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:16:22.156328 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:22.156284 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" podUID="8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:16:22.200707 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:22.200678 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:22.308446 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:22.308352 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rrwcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:16:22.308760 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:22.308731 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" podUID="80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:16:24.113442 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:24.113407 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:25.691325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:25.691291 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs"] Oct 30 06:16:25.691792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:25.691535 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" podUID="186bbd33-30f6-4334-b87a-fa151cc6cd61" containerName="maas-api" containerID="cri-o://121cfe3d00fb55bab111dbb5f1b9178515ca966aed7281d6cadb05c9c7df82cd" gracePeriod=30 Oct 30 06:16:25.791992 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:25.791959 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:26.015689 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.015659 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:16:26.106975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.106946 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186bbd33-30f6-4334-b87a-fa151cc6cd61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/186bbd33-30f6-4334-b87a-fa151cc6cd61-metrics-certs\") pod \"186bbd33-30f6-4334-b87a-fa151cc6cd61\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " Oct 30 06:16:26.107164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.107030 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186bbd33-30f6-4334-b87a-fa151cc6cd61-kube-api-access-bzvgx\" (UniqueName: \"kubernetes.io/projected/186bbd33-30f6-4334-b87a-fa151cc6cd61-kube-api-access-bzvgx\") pod \"186bbd33-30f6-4334-b87a-fa151cc6cd61\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " Oct 30 06:16:26.107164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.107140 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186bbd33-30f6-4334-b87a-fa151cc6cd61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186bbd33-30f6-4334-b87a-fa151cc6cd61-maas-api-tls\") pod \"186bbd33-30f6-4334-b87a-fa151cc6cd61\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " Oct 30 06:16:26.107353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.107291 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/186bbd33-30f6-4334-b87a-fa151cc6cd61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186bbd33-30f6-4334-b87a-fa151cc6cd61-openshift-service-ca\") pod \"186bbd33-30f6-4334-b87a-fa151cc6cd61\" (UID: \"186bbd33-30f6-4334-b87a-fa151cc6cd61\") " Oct 30 06:16:26.107895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.107866 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/186bbd33-30f6-4334-b87a-fa151cc6cd61-openshift-service-ca" pod "186bbd33-30f6-4334-b87a-fa151cc6cd61" (UID: "186bbd33-30f6-4334-b87a-fa151cc6cd61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:26.110827 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.110799 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186bbd33-30f6-4334-b87a-fa151cc6cd61-kube-api-access-bzvgx" pod "186bbd33-30f6-4334-b87a-fa151cc6cd61" (UID: "186bbd33-30f6-4334-b87a-fa151cc6cd61"). InnerVolumeSpecName "kube-api-access-bzvgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:26.111357 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.111332 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186bbd33-30f6-4334-b87a-fa151cc6cd61-maas-api-tls" pod "186bbd33-30f6-4334-b87a-fa151cc6cd61" (UID: "186bbd33-30f6-4334-b87a-fa151cc6cd61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:26.115601 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.115579 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186bbd33-30f6-4334-b87a-fa151cc6cd61-metrics-certs" pod "186bbd33-30f6-4334-b87a-fa151cc6cd61" (UID: "186bbd33-30f6-4334-b87a-fa151cc6cd61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:26.210011 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.209973 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/186bbd33-30f6-4334-b87a-fa151cc6cd61-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:16:26.210011 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.210014 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/186bbd33-30f6-4334-b87a-fa151cc6cd61-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:16:26.210342 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.210032 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/186bbd33-30f6-4334-b87a-fa151cc6cd61-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:16:26.210342 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.210049 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzvgx\" (UniqueName: \"kubernetes.io/projected/186bbd33-30f6-4334-b87a-fa151cc6cd61-kube-api-access-bzvgx\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:16:26.284309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.283373 6999 generic.go:356] "Generic (PLEG): container finished" podID="186bbd33-30f6-4334-b87a-fa151cc6cd61" containerID="121cfe3d00fb55bab111dbb5f1b9178515ca966aed7281d6cadb05c9c7df82cd" exitCode=0 Oct 30 06:16:26.284309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.283545 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" event={"ID":"186bbd33-30f6-4334-b87a-fa151cc6cd61","Type":"ContainerDied","Data":"121cfe3d00fb55bab111dbb5f1b9178515ca966aed7281d6cadb05c9c7df82cd"} Oct 30 06:16:26.284309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.283613 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" event={"ID":"186bbd33-30f6-4334-b87a-fa151cc6cd61","Type":"ContainerDied","Data":"fc789944b432b7c4a0bc2ca4461999391f218baa00c6d4353fe9c161b0c8d29e"} Oct 30 06:16:26.284309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.283644 6999 scope.go:122] "RemoveContainer" containerID="121cfe3d00fb55bab111dbb5f1b9178515ca966aed7281d6cadb05c9c7df82cd" Oct 30 06:16:26.284309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.283810 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs" Oct 30 06:16:26.302780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.302728 6999 scope.go:122] "RemoveContainer" containerID="121cfe3d00fb55bab111dbb5f1b9178515ca966aed7281d6cadb05c9c7df82cd" Oct 30 06:16:26.303733 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:16:26.303702 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"121cfe3d00fb55bab111dbb5f1b9178515ca966aed7281d6cadb05c9c7df82cd\": container with ID starting with 121cfe3d00fb55bab111dbb5f1b9178515ca966aed7281d6cadb05c9c7df82cd not found: ID does not exist" containerID="121cfe3d00fb55bab111dbb5f1b9178515ca966aed7281d6cadb05c9c7df82cd" Oct 30 06:16:26.303842 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.303747 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"121cfe3d00fb55bab111dbb5f1b9178515ca966aed7281d6cadb05c9c7df82cd"} err="failed to get container status \"121cfe3d00fb55bab111dbb5f1b9178515ca966aed7281d6cadb05c9c7df82cd\": rpc error: code = NotFound desc = could not find container \"121cfe3d00fb55bab111dbb5f1b9178515ca966aed7281d6cadb05c9c7df82cd\": container with ID starting with 121cfe3d00fb55bab111dbb5f1b9178515ca966aed7281d6cadb05c9c7df82cd not found: ID does not exist" Oct 30 06:16:26.320801 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.320769 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:26.341688 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.340937 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs"] Oct 30 06:16:26.351156 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.351123 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-fe108e-687ccbbf88-ghfjs"] Oct 30 06:16:26.492884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:26.492850 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:27.066447 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:27.066416 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:27.609679 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:27.609642 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186bbd33-30f6-4334-b87a-fa151cc6cd61" path="/var/lib/kubelet/pods/186bbd33-30f6-4334-b87a-fa151cc6cd61/volumes" Oct 30 06:16:28.328152 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:28.328119 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:30.236279 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:30.235453 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:31.040420 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:31.040389 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt"] Oct 30 06:16:31.052567 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:31.052542 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca9e7e78-zsqjt"] Oct 30 06:16:31.610850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:31.610821 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cbecf98-3905-4212-994c-447b3cae6827" path="/var/lib/kubelet/pods/1cbecf98-3905-4212-994c-447b3cae6827/volumes" Oct 30 06:16:32.016350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:32.016258 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:33.342462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.342430 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj"] Oct 30 06:16:33.343006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.342715 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" podUID="85f3764f-43fd-4956-b4de-c73fc1285683" containerName="maas-api" containerID="cri-o://319a801d9e3782bf40c1e54397a76a6c0d2053d571673f9954aee6e2667ad91c" gracePeriod=30 Oct 30 06:16:33.431804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.431719 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:33.707919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.707888 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:16:33.825144 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.825110 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-metrics-certs\") pod \"85f3764f-43fd-4956-b4de-c73fc1285683\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " Oct 30 06:16:33.825351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.825188 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f3764f-43fd-4956-b4de-c73fc1285683-kube-api-access-8ljkb\" (UniqueName: \"kubernetes.io/projected/85f3764f-43fd-4956-b4de-c73fc1285683-kube-api-access-8ljkb\") pod \"85f3764f-43fd-4956-b4de-c73fc1285683\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " Oct 30 06:16:33.825351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.825256 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f3764f-43fd-4956-b4de-c73fc1285683-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f3764f-43fd-4956-b4de-c73fc1285683-openshift-service-ca\") pod \"85f3764f-43fd-4956-b4de-c73fc1285683\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " Oct 30 06:16:33.825490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.825411 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-maas-api-tls\") pod \"85f3764f-43fd-4956-b4de-c73fc1285683\" (UID: \"85f3764f-43fd-4956-b4de-c73fc1285683\") " Oct 30 06:16:33.825662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.825638 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f3764f-43fd-4956-b4de-c73fc1285683-openshift-service-ca" pod "85f3764f-43fd-4956-b4de-c73fc1285683" (UID: "85f3764f-43fd-4956-b4de-c73fc1285683"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:33.825979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.825957 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f3764f-43fd-4956-b4de-c73fc1285683-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:16:33.827716 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.827690 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-metrics-certs" pod "85f3764f-43fd-4956-b4de-c73fc1285683" (UID: "85f3764f-43fd-4956-b4de-c73fc1285683"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:33.828161 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.828141 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-maas-api-tls" pod "85f3764f-43fd-4956-b4de-c73fc1285683" (UID: "85f3764f-43fd-4956-b4de-c73fc1285683"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:33.828238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.828198 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f3764f-43fd-4956-b4de-c73fc1285683-kube-api-access-8ljkb" pod "85f3764f-43fd-4956-b4de-c73fc1285683" (UID: "85f3764f-43fd-4956-b4de-c73fc1285683"). InnerVolumeSpecName "kube-api-access-8ljkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:33.927246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.927121 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ljkb\" (UniqueName: \"kubernetes.io/projected/85f3764f-43fd-4956-b4de-c73fc1285683-kube-api-access-8ljkb\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:16:33.927246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.927153 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:16:33.927246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.927163 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f3764f-43fd-4956-b4de-c73fc1285683-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:16:33.994403 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:33.994370 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:34.141967 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:34.141912 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:34.159026 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:16:34.158992 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:16:34.338061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:34.337970 6999 generic.go:356] "Generic (PLEG): container finished" podID="85f3764f-43fd-4956-b4de-c73fc1285683" containerID="319a801d9e3782bf40c1e54397a76a6c0d2053d571673f9954aee6e2667ad91c" exitCode=0 Oct 30 06:16:34.338061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:34.338014 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" Oct 30 06:16:34.338300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:34.338089 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" event={"ID":"85f3764f-43fd-4956-b4de-c73fc1285683","Type":"ContainerDied","Data":"319a801d9e3782bf40c1e54397a76a6c0d2053d571673f9954aee6e2667ad91c"} Oct 30 06:16:34.338300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:34.338154 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj" event={"ID":"85f3764f-43fd-4956-b4de-c73fc1285683","Type":"ContainerDied","Data":"daa9b4c95f58f182c33af67260f621be73574f16366ee69a233001d81cf0aa4d"} Oct 30 06:16:34.338300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:34.338187 6999 scope.go:122] "RemoveContainer" containerID="319a801d9e3782bf40c1e54397a76a6c0d2053d571673f9954aee6e2667ad91c" Oct 30 06:16:34.349175 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:34.348978 6999 scope.go:122] "RemoveContainer" containerID="319a801d9e3782bf40c1e54397a76a6c0d2053d571673f9954aee6e2667ad91c" Oct 30 06:16:34.349487 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:16:34.349309 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319a801d9e3782bf40c1e54397a76a6c0d2053d571673f9954aee6e2667ad91c\": container with ID starting with 319a801d9e3782bf40c1e54397a76a6c0d2053d571673f9954aee6e2667ad91c not found: ID does not exist" containerID="319a801d9e3782bf40c1e54397a76a6c0d2053d571673f9954aee6e2667ad91c" Oct 30 06:16:34.349487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:34.349335 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319a801d9e3782bf40c1e54397a76a6c0d2053d571673f9954aee6e2667ad91c"} err="failed to get container status \"319a801d9e3782bf40c1e54397a76a6c0d2053d571673f9954aee6e2667ad91c\": rpc error: code = NotFound desc = could not find container \"319a801d9e3782bf40c1e54397a76a6c0d2053d571673f9954aee6e2667ad91c\": container with ID starting with 319a801d9e3782bf40c1e54397a76a6c0d2053d571673f9954aee6e2667ad91c not found: ID does not exist" Oct 30 06:16:34.389858 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:34.389834 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj"] Oct 30 06:16:34.401802 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:34.401776 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c15d3a-6c4f6d69fc-n92rj"] Oct 30 06:16:34.732627 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:34.732595 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:34.870501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:34.870464 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:35.132344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:35.132296 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:35.608774 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:35.608738 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f3764f-43fd-4956-b4de-c73fc1285683" path="/var/lib/kubelet/pods/85f3764f-43fd-4956-b4de-c73fc1285683/volumes" Oct 30 06:16:36.105743 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:36.105654 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:38.131867 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:38.131833 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:39.624264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:39.615546 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:39.992357 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:39.992279 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:40.435201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:40.435159 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:41.601198 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:41.601162 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:41.846150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:41.846116 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:42.108498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:42.108442 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:42.278109 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:42.278071 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:42.450773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:42.450686 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:43.821754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:43.821726 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:43.969307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:43.969279 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:44.314794 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:44.314710 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:44.469976 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:44.469941 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:45.945091 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:45.945044 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:46.452680 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:46.452651 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:46.630330 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:46.630292 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:47.514056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:47.514021 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:47.781425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:47.781342 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:48.037453 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.037380 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:48.283855 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.283823 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.443741 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.449152 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5"] Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450096 6999 cpu_manager.go:414] "containerMap: removing container" podUID="85f3764f-43fd-4956-b4de-c73fc1285683" containerName="maas-api" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450117 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f3764f-43fd-4956-b4de-c73fc1285683" containerName="maas-api" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450135 6999 cpu_manager.go:414] "containerMap: removing container" podUID="1bcbb1eb-21c8-4b95-a209-8d94d54f719d" containerName="maas-api" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450142 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bcbb1eb-21c8-4b95-a209-8d94d54f719d" containerName="maas-api" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450202 6999 cpu_manager.go:414] "containerMap: removing container" podUID="ff79aff8-16c9-4edf-8b16-81ef85272ae5" containerName="payload-pre-processing" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450228 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff79aff8-16c9-4edf-8b16-81ef85272ae5" containerName="payload-pre-processing" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450254 6999 cpu_manager.go:414] "containerMap: removing container" podUID="12365d0d-c44f-488e-9362-d5f74d4de585" containerName="maas-api" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450261 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12365d0d-c44f-488e-9362-d5f74d4de585" containerName="maas-api" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450280 6999 cpu_manager.go:414] "containerMap: removing container" podUID="c4989e46-f4b8-4113-aea1-66994ea9c6c6" containerName="istio-proxy" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450290 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4989e46-f4b8-4113-aea1-66994ea9c6c6" containerName="istio-proxy" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450299 6999 cpu_manager.go:414] "containerMap: removing container" podUID="186bbd33-30f6-4334-b87a-fa151cc6cd61" containerName="maas-api" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450306 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186bbd33-30f6-4334-b87a-fa151cc6cd61" containerName="maas-api" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450447 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff79aff8-16c9-4edf-8b16-81ef85272ae5" containerName="payload-pre-processing" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450466 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="12365d0d-c44f-488e-9362-d5f74d4de585" containerName="maas-api" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450478 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4989e46-f4b8-4113-aea1-66994ea9c6c6" containerName="istio-proxy" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450488 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="186bbd33-30f6-4334-b87a-fa151cc6cd61" containerName="maas-api" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450500 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f3764f-43fd-4956-b4de-c73fc1285683" containerName="maas-api" Oct 30 06:16:48.451292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.450512 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bcbb1eb-21c8-4b95-a209-8d94d54f719d" containerName="maas-api" Oct 30 06:16:48.454392 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.454371 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" Oct 30 06:16:48.479314 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.479287 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5"] Oct 30 06:16:48.481638 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.481612 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-33c99c\"" Oct 30 06:16:48.481816 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.481701 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-33c99c-dockercfg-plrl5\"" Oct 30 06:16:48.552660 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.552622 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr8x5\" (UniqueName: \"kubernetes.io/projected/2d2f6047-f480-493a-9726-0e078ca0964e-kube-api-access-xr8x5\") pod \"payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5\" (UID: \"2d2f6047-f480-493a-9726-0e078ca0964e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" Oct 30 06:16:48.553121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.552698 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2f6047-f480-493a-9726-0e078ca0964e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5\" (UID: \"2d2f6047-f480-493a-9726-0e078ca0964e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" Oct 30 06:16:48.612051 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.612011 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:48.654394 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.654362 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr8x5\" (UniqueName: \"kubernetes.io/projected/2d2f6047-f480-493a-9726-0e078ca0964e-kube-api-access-xr8x5\") pod \"payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5\" (UID: \"2d2f6047-f480-493a-9726-0e078ca0964e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" Oct 30 06:16:48.654649 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.654421 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2f6047-f480-493a-9726-0e078ca0964e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5\" (UID: \"2d2f6047-f480-493a-9726-0e078ca0964e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" Oct 30 06:16:48.655340 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.655314 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2f6047-f480-493a-9726-0e078ca0964e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5\" (UID: \"2d2f6047-f480-493a-9726-0e078ca0964e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" Oct 30 06:16:48.678366 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.678337 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr8x5\" (UniqueName: \"kubernetes.io/projected/2d2f6047-f480-493a-9726-0e078ca0964e-kube-api-access-xr8x5\") pod \"payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5\" (UID: \"2d2f6047-f480-493a-9726-0e078ca0964e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" Oct 30 06:16:48.780925 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.780841 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:48.781088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.781072 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" Oct 30 06:16:48.923923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:48.923450 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:49.118471 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:16:49.118442 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2f6047_f480_493a_9726_0e078ca0964e.slice/crio-f95838aac528ad97ae429c46cedfce2925f6bc4bb75e2077df1810f4fb17c461 WatchSource:0}: Error finding container f95838aac528ad97ae429c46cedfce2925f6bc4bb75e2077df1810f4fb17c461: Status 404 returned error can't find the container with id f95838aac528ad97ae429c46cedfce2925f6bc4bb75e2077df1810f4fb17c461 Oct 30 06:16:49.121452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:49.121424 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5"] Oct 30 06:16:49.463226 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:49.463001 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:49.507239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:49.494732 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" event={"ID":"2d2f6047-f480-493a-9726-0e078ca0964e","Type":"ContainerStarted","Data":"671c4957a65be2ec8255282ea6f4a897d349f94ad663d21342acf84eaeb108fb"} Oct 30 06:16:49.507239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:49.494805 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" event={"ID":"2d2f6047-f480-493a-9726-0e078ca0964e","Type":"ContainerStarted","Data":"f95838aac528ad97ae429c46cedfce2925f6bc4bb75e2077df1810f4fb17c461"} Oct 30 06:16:49.507239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:49.495995 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" Oct 30 06:16:49.566324 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:49.566255 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" podStartSLOduration=1.566235238 podStartE2EDuration="1.566235238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:49.563916181 +0000 UTC m=+1376.072936069" watchObservedRunningTime="2026-10-30 06:16:49.566235238 +0000 UTC m=+1376.075255126" Oct 30 06:16:49.620566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:49.620537 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:50.304662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.304574 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:50.399047 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.399017 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82"] Oct 30 06:16:50.403011 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.402988 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" Oct 30 06:16:50.419908 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.419887 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-584c9d\"" Oct 30 06:16:50.432700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.432656 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-584c9d-dockercfg-w5mlr\"" Oct 30 06:16:50.439057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.437523 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82"] Oct 30 06:16:50.449638 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.449618 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78"] Oct 30 06:16:50.453720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.453701 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" Oct 30 06:16:50.466773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.466748 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:50.467265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.467121 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78"] Oct 30 06:16:50.482536 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.482514 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e6de368-45ad-4356-8c54-66e69475abe2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82\" (UID: \"2e6de368-45ad-4356-8c54-66e69475abe2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" Oct 30 06:16:50.482851 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.482825 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nzbx\" (UniqueName: \"kubernetes.io/projected/2e6de368-45ad-4356-8c54-66e69475abe2-kube-api-access-9nzbx\") pod \"payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82\" (UID: \"2e6de368-45ad-4356-8c54-66e69475abe2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" Oct 30 06:16:50.590319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.588165 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e6de368-45ad-4356-8c54-66e69475abe2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82\" (UID: \"2e6de368-45ad-4356-8c54-66e69475abe2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" Oct 30 06:16:50.590319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.588283 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e85bdffe-82ea-4f28-870b-59983e2ce7df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78\" (UID: \"e85bdffe-82ea-4f28-870b-59983e2ce7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" Oct 30 06:16:50.590319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.588332 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbwvr\" (UniqueName: \"kubernetes.io/projected/e85bdffe-82ea-4f28-870b-59983e2ce7df-kube-api-access-vbwvr\") pod \"payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78\" (UID: \"e85bdffe-82ea-4f28-870b-59983e2ce7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" Oct 30 06:16:50.590319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.588624 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzbx\" (UniqueName: \"kubernetes.io/projected/2e6de368-45ad-4356-8c54-66e69475abe2-kube-api-access-9nzbx\") pod \"payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82\" (UID: \"2e6de368-45ad-4356-8c54-66e69475abe2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" Oct 30 06:16:50.590319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.589772 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e6de368-45ad-4356-8c54-66e69475abe2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82\" (UID: \"2e6de368-45ad-4356-8c54-66e69475abe2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" Oct 30 06:16:50.613199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.613169 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzbx\" (UniqueName: \"kubernetes.io/projected/2e6de368-45ad-4356-8c54-66e69475abe2-kube-api-access-9nzbx\") pod \"payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82\" (UID: \"2e6de368-45ad-4356-8c54-66e69475abe2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" Oct 30 06:16:50.691288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.689632 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e85bdffe-82ea-4f28-870b-59983e2ce7df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78\" (UID: \"e85bdffe-82ea-4f28-870b-59983e2ce7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" Oct 30 06:16:50.691288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.689693 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwvr\" (UniqueName: \"kubernetes.io/projected/e85bdffe-82ea-4f28-870b-59983e2ce7df-kube-api-access-vbwvr\") pod \"payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78\" (UID: \"e85bdffe-82ea-4f28-870b-59983e2ce7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" Oct 30 06:16:50.691288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.691145 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e85bdffe-82ea-4f28-870b-59983e2ce7df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78\" (UID: \"e85bdffe-82ea-4f28-870b-59983e2ce7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" Oct 30 06:16:50.713306 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.713278 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbwvr\" (UniqueName: \"kubernetes.io/projected/e85bdffe-82ea-4f28-870b-59983e2ce7df-kube-api-access-vbwvr\") pod \"payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78\" (UID: \"e85bdffe-82ea-4f28-870b-59983e2ce7df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" Oct 30 06:16:50.716726 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.716560 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" Oct 30 06:16:50.777088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:50.775792 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" Oct 30 06:16:51.027551 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:51.027529 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82"] Oct 30 06:16:51.028599 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:16:51.028571 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6de368_45ad_4356_8c54_66e69475abe2.slice/crio-96e404388e1a61a93d543b22cd411ca05b4bf6b1c398455bdc16f80f414d85c9 WatchSource:0}: Error finding container 96e404388e1a61a93d543b22cd411ca05b4bf6b1c398455bdc16f80f414d85c9: Status 404 returned error can't find the container with id 96e404388e1a61a93d543b22cd411ca05b4bf6b1c398455bdc16f80f414d85c9 Oct 30 06:16:51.123880 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:51.123857 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78"] Oct 30 06:16:51.125056 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:16:51.125035 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85bdffe_82ea_4f28_870b_59983e2ce7df.slice/crio-3bdd121ecaffa98d4a9c4017f56c4a32e44e1a3ec6c1b47c71294d0367e94cf1 WatchSource:0}: Error finding container 3bdd121ecaffa98d4a9c4017f56c4a32e44e1a3ec6c1b47c71294d0367e94cf1: Status 404 returned error can't find the container with id 3bdd121ecaffa98d4a9c4017f56c4a32e44e1a3ec6c1b47c71294d0367e94cf1 Oct 30 06:16:51.510140 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:51.510053 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" event={"ID":"e85bdffe-82ea-4f28-870b-59983e2ce7df","Type":"ContainerStarted","Data":"941a023f9a8ec8c89a69a8fa5a5a4a2dd1162fc8d96af10f48923d71130ed0b0"} Oct 30 06:16:51.510140 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:51.510136 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" Oct 30 06:16:51.510419 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:51.510160 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" event={"ID":"e85bdffe-82ea-4f28-870b-59983e2ce7df","Type":"ContainerStarted","Data":"3bdd121ecaffa98d4a9c4017f56c4a32e44e1a3ec6c1b47c71294d0367e94cf1"} Oct 30 06:16:51.511912 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:51.511880 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" event={"ID":"2e6de368-45ad-4356-8c54-66e69475abe2","Type":"ContainerStarted","Data":"b98d983058e6caa16a0fe0f9d0defbf5f03915babe95a7eae4001f7c1b5edb6d"} Oct 30 06:16:51.512027 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:51.511946 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" Oct 30 06:16:51.512027 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:51.511971 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" event={"ID":"2e6de368-45ad-4356-8c54-66e69475abe2","Type":"ContainerStarted","Data":"96e404388e1a61a93d543b22cd411ca05b4bf6b1c398455bdc16f80f414d85c9"} Oct 30 06:16:51.552043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:51.552000 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" podStartSLOduration=1.551987482 podStartE2EDuration="1.551987482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:51.549826983 +0000 UTC m=+1378.058846894" watchObservedRunningTime="2026-10-30 06:16:51.551987482 +0000 UTC m=+1378.061007396" Oct 30 06:16:51.591772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:51.591728 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" podStartSLOduration=1.591715385 podStartE2EDuration="1.591715385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:51.589843253 +0000 UTC m=+1378.098863141" watchObservedRunningTime="2026-10-30 06:16:51.591715385 +0000 UTC m=+1378.100735272" Oct 30 06:16:52.359200 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:52.359152 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:53.233311 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:53.233280 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw"] Oct 30 06:16:53.392687 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:53.392657 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:53.677232 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:53.677182 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:53.831459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:53.831419 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:53.985155 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:53.985055 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:54.009387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:54.004571 6999 scope.go:122] "RemoveContainer" containerID="4879cd429720feca39972993aac143864b94d76f02432c70c2b15d8d2e91dfc8" Oct 30 06:16:54.678553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:54.678527 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:55.034867 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:55.034782 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt"] Oct 30 06:16:55.066991 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:55.066949 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81bff280-nspmt"] Oct 30 06:16:55.491180 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:55.491151 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:55.609646 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:55.609615 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e06707a7-4643-4c6a-b18e-5bcd55e16d37" path="/var/lib/kubelet/pods/e06707a7-4643-4c6a-b18e-5bcd55e16d37/volumes" Oct 30 06:16:55.638764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:55.638731 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:56.559073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:56.559045 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:58.327024 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:58.326994 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:58.539726 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:58.539688 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" podUID="2342e52a-f07a-4069-a24b-e476995a0fbd" containerName="payload-processing" containerID="cri-o://de19bed6965ddb94f105e00c74f43c4bfb4e043d96733445884526564aa2dae2" gracePeriod=25 Oct 30 06:16:58.815025 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:58.815004 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" Oct 30 06:16:58.922720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:58.922648 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2342e52a-f07a-4069-a24b-e476995a0fbd-kube-api-access-c4j56\" (UniqueName: \"kubernetes.io/projected/2342e52a-f07a-4069-a24b-e476995a0fbd-kube-api-access-c4j56\") pod \"2342e52a-f07a-4069-a24b-e476995a0fbd\" (UID: \"2342e52a-f07a-4069-a24b-e476995a0fbd\") " Oct 30 06:16:58.922720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:58.922686 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2342e52a-f07a-4069-a24b-e476995a0fbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2342e52a-f07a-4069-a24b-e476995a0fbd-plugins-config-volume\") pod \"2342e52a-f07a-4069-a24b-e476995a0fbd\" (UID: \"2342e52a-f07a-4069-a24b-e476995a0fbd\") " Oct 30 06:16:58.923096 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:58.923077 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2342e52a-f07a-4069-a24b-e476995a0fbd-plugins-config-volume" pod "2342e52a-f07a-4069-a24b-e476995a0fbd" (UID: "2342e52a-f07a-4069-a24b-e476995a0fbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:58.923308 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:58.923287 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2342e52a-f07a-4069-a24b-e476995a0fbd-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:16:58.924905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:58.924889 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2342e52a-f07a-4069-a24b-e476995a0fbd-kube-api-access-c4j56" pod "2342e52a-f07a-4069-a24b-e476995a0fbd" (UID: "2342e52a-f07a-4069-a24b-e476995a0fbd"). InnerVolumeSpecName "kube-api-access-c4j56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:59.024606 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.024572 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4j56\" (UniqueName: \"kubernetes.io/projected/2342e52a-f07a-4069-a24b-e476995a0fbd-kube-api-access-c4j56\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:16:59.548167 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.548135 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:16:59.580763 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.580736 6999 generic.go:356] "Generic (PLEG): container finished" podID="2342e52a-f07a-4069-a24b-e476995a0fbd" containerID="de19bed6965ddb94f105e00c74f43c4bfb4e043d96733445884526564aa2dae2" exitCode=0 Oct 30 06:16:59.580928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.580777 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" Oct 30 06:16:59.580928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.580819 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" event={"ID":"2342e52a-f07a-4069-a24b-e476995a0fbd","Type":"ContainerDied","Data":"de19bed6965ddb94f105e00c74f43c4bfb4e043d96733445884526564aa2dae2"} Oct 30 06:16:59.580928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.580876 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw" event={"ID":"2342e52a-f07a-4069-a24b-e476995a0fbd","Type":"ContainerDied","Data":"7de06642fde544dccf58dee275be77043eba17dfb360bb5eb8eb07c4874aa3b3"} Oct 30 06:16:59.580928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.580909 6999 scope.go:122] "RemoveContainer" containerID="de19bed6965ddb94f105e00c74f43c4bfb4e043d96733445884526564aa2dae2" Oct 30 06:16:59.591705 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.591691 6999 scope.go:122] "RemoveContainer" containerID="de19bed6965ddb94f105e00c74f43c4bfb4e043d96733445884526564aa2dae2" Oct 30 06:16:59.591941 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:16:59.591923 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de19bed6965ddb94f105e00c74f43c4bfb4e043d96733445884526564aa2dae2\": container with ID starting with de19bed6965ddb94f105e00c74f43c4bfb4e043d96733445884526564aa2dae2 not found: ID does not exist" containerID="de19bed6965ddb94f105e00c74f43c4bfb4e043d96733445884526564aa2dae2" Oct 30 06:16:59.592008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.591951 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de19bed6965ddb94f105e00c74f43c4bfb4e043d96733445884526564aa2dae2"} err="failed to get container status \"de19bed6965ddb94f105e00c74f43c4bfb4e043d96733445884526564aa2dae2\": rpc error: code = NotFound desc = could not find container \"de19bed6965ddb94f105e00c74f43c4bfb4e043d96733445884526564aa2dae2\": container with ID starting with de19bed6965ddb94f105e00c74f43c4bfb4e043d96733445884526564aa2dae2 not found: ID does not exist" Oct 30 06:16:59.632882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.632856 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw"] Oct 30 06:16:59.644670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.644642 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f7a606-6f7fd6cf96-pcpbw"] Oct 30 06:16:59.961267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.961238 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh"] Oct 30 06:16:59.961867 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.961846 6999 cpu_manager.go:414] "containerMap: removing container" podUID="2342e52a-f07a-4069-a24b-e476995a0fbd" containerName="payload-processing" Oct 30 06:16:59.962005 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.961868 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2342e52a-f07a-4069-a24b-e476995a0fbd" containerName="payload-processing" Oct 30 06:16:59.962083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.962015 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="2342e52a-f07a-4069-a24b-e476995a0fbd" containerName="payload-processing" Oct 30 06:16:59.966942 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.966924 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:16:59.987011 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.986985 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-33c99c-serving-cert\"" Oct 30 06:16:59.991077 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:16:59.991055 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh"] Oct 30 06:17:00.036894 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.036867 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngpgh\" (UniqueName: \"kubernetes.io/projected/ace3fa7d-d193-4184-8850-c4741edd37b3-kube-api-access-ngpgh\") pod \"maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:00.037082 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.036912 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace3fa7d-d193-4184-8850-c4741edd37b3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:00.037082 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.036934 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:00.037247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.037208 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-metrics-certs\") pod \"maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:00.092503 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.092473 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:00.138662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.138636 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-metrics-certs\") pod \"maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:00.138801 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.138787 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngpgh\" (UniqueName: \"kubernetes.io/projected/ace3fa7d-d193-4184-8850-c4741edd37b3-kube-api-access-ngpgh\") pod \"maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:00.138884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.138835 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace3fa7d-d193-4184-8850-c4741edd37b3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:00.138884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.138861 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:00.139122 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:17:00.139102 6999 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-serving-cert: secret "maas-api-e2e-ipp-a-33c99c-serving-cert" not found Oct 30 06:17:00.139270 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:17:00.139184 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-maas-api-tls podName:ace3fa7d-d193-4184-8850-c4741edd37b3 nodeName:}" failed. No retries permitted until 2026-10-30 06:17:00.639163628 +0000 UTC m=+1387.148183502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-maas-api-tls") pod "maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" (UID: "ace3fa7d-d193-4184-8850-c4741edd37b3") : secret "maas-api-e2e-ipp-a-33c99c-serving-cert" not found Oct 30 06:17:00.139791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.139768 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace3fa7d-d193-4184-8850-c4741edd37b3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:00.141401 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.141374 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-metrics-certs\") pod \"maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:00.175718 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.169038 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngpgh\" (UniqueName: \"kubernetes.io/projected/ace3fa7d-d193-4184-8850-c4741edd37b3-kube-api-access-ngpgh\") pod \"maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:00.240113 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.240021 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:00.446664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.446633 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:00.644014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.643986 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:00.646543 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.646524 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:00.886575 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:00.886541 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:01.308419 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:01.308389 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh"] Oct 30 06:17:01.309598 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:17:01.309572 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace3fa7d_d193_4184_8850_c4741edd37b3.slice/crio-0ac71a62e4b628ae9c8ad854f60a73a2e038b01982166ac00f52151fee85f393 WatchSource:0}: Error finding container 0ac71a62e4b628ae9c8ad854f60a73a2e038b01982166ac00f52151fee85f393: Status 404 returned error can't find the container with id 0ac71a62e4b628ae9c8ad854f60a73a2e038b01982166ac00f52151fee85f393 Oct 30 06:17:01.476910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:01.476880 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:01.515725 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:01.515703 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" Oct 30 06:17:01.597565 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:01.597481 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" event={"ID":"ace3fa7d-d193-4184-8850-c4741edd37b3","Type":"ContainerStarted","Data":"0ac71a62e4b628ae9c8ad854f60a73a2e038b01982166ac00f52151fee85f393"} Oct 30 06:17:01.610674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:01.610636 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2342e52a-f07a-4069-a24b-e476995a0fbd" path="/var/lib/kubelet/pods/2342e52a-f07a-4069-a24b-e476995a0fbd/volumes" Oct 30 06:17:02.021538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:02.021494 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:02.535049 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:02.535024 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" Oct 30 06:17:02.535269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:02.535088 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" Oct 30 06:17:02.607428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:02.607392 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" event={"ID":"ace3fa7d-d193-4184-8850-c4741edd37b3","Type":"ContainerStarted","Data":"62c2316cbe7e39f7ac0b2e8ab72f2a49ce620d0611e73e31641449241fd1ac8c"} Oct 30 06:17:02.607633 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:02.607540 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:02.663340 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:02.663287 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" podStartSLOduration=3.185157718 podStartE2EDuration="3.663272248s" totalImagesPullingTime="478.11453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:02.660065791 +0000 UTC m=+1389.169085680" watchObservedRunningTime="2026-10-30 06:17:02.663272248 +0000 UTC m=+1389.172292138" Oct 30 06:17:03.103601 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.103568 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:03.652258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.652209 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf"] Oct 30 06:17:03.656427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.656411 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:03.673373 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.673233 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-584c9d-serving-cert\"" Oct 30 06:17:03.688329 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.688300 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf"] Oct 30 06:17:03.771112 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.771078 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:03.804619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.804590 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:03.804793 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.804777 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhvxx\" (UniqueName: \"kubernetes.io/projected/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-kube-api-access-zhvxx\") pod \"maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:03.804858 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.804813 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-metrics-certs\") pod \"maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:03.804858 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.804847 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:03.906179 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.906091 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhvxx\" (UniqueName: \"kubernetes.io/projected/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-kube-api-access-zhvxx\") pod \"maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:03.906179 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.906146 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-metrics-certs\") pod \"maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:03.906450 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.906187 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:03.906450 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.906254 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:03.906968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.906940 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:03.908854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.908837 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:03.908939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.908889 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-metrics-certs\") pod \"maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:03.956899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.956868 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhvxx\" (UniqueName: \"kubernetes.io/projected/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-kube-api-access-zhvxx\") pod \"maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:03.970700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.970670 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:03.991009 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:03.990973 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:04.142949 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:17:04.142915 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:17:04.175723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:04.175612 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:04.289250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:04.289204 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf"] Oct 30 06:17:04.290340 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:17:04.290316 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3786e3f7_54a2_44b9_a7b0_49c5e333f0f0.slice/crio-0e1bfbff14db0213dda2844da5b9b0c25cb78bb95cbdd9298169b7f08156de86 WatchSource:0}: Error finding container 0e1bfbff14db0213dda2844da5b9b0c25cb78bb95cbdd9298169b7f08156de86: Status 404 returned error can't find the container with id 0e1bfbff14db0213dda2844da5b9b0c25cb78bb95cbdd9298169b7f08156de86 Oct 30 06:17:04.320469 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:04.320445 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:04.623233 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:04.623173 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" event={"ID":"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0","Type":"ContainerStarted","Data":"0e1bfbff14db0213dda2844da5b9b0c25cb78bb95cbdd9298169b7f08156de86"} Oct 30 06:17:05.036276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:05.036144 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs"] Oct 30 06:17:05.047564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:05.047537 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bb09bcb-cmnfs"] Oct 30 06:17:05.608904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:05.608871 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15ea8ec2-12f7-491b-97ba-deedcd92f3ef" path="/var/lib/kubelet/pods/15ea8ec2-12f7-491b-97ba-deedcd92f3ef/volumes" Oct 30 06:17:05.630670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:05.630641 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" event={"ID":"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0","Type":"ContainerStarted","Data":"44336a6ef2dd02899a56a4993373ad53e7c77ea68bb30ee9e8433d6108ee6d6e"} Oct 30 06:17:05.630860 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:05.630842 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:05.676127 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:05.676083 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" podStartSLOduration=2.287189059 podStartE2EDuration="2.676072694s" totalImagesPullingTime="388.883635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:05.675691643 +0000 UTC m=+1392.184711530" watchObservedRunningTime="2026-10-30 06:17:05.676072694 +0000 UTC m=+1392.185092584" Oct 30 06:17:05.742326 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:05.742301 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:07.627170 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:07.627142 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:08.272493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:08.272460 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:08.618636 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:08.618615 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:17:08.755908 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:08.755878 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:09.287308 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.287265 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:09.290416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.288456 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk"] Oct 30 06:17:09.292564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.292544 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:09.326630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.326582 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk"] Oct 30 06:17:09.366468 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.366438 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0fab6d-serving-cert\"" Oct 30 06:17:09.420239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.396167 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9wt8\" (UniqueName: \"kubernetes.io/projected/179d1033-3d70-4cca-a7cc-0a6bc3223623-kube-api-access-z9wt8\") pod \"maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:09.420239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.396316 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-maas-api-tls\") pod \"maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:09.420239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.396417 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/179d1033-3d70-4cca-a7cc-0a6bc3223623-openshift-service-ca\") pod \"maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:09.420239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.396445 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-metrics-certs\") pod \"maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:09.482235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.482186 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:09.497427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.497394 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-maas-api-tls\") pod \"maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:09.497643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.497489 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/179d1033-3d70-4cca-a7cc-0a6bc3223623-openshift-service-ca\") pod \"maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:09.497643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.497530 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-metrics-certs\") pod \"maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:09.497643 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:17:09.497572 6999 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-serving-cert: secret "maas-api-e2e-api-a-0fab6d-serving-cert" not found Oct 30 06:17:09.497643 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:17:09.497636 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-maas-api-tls podName:179d1033-3d70-4cca-a7cc-0a6bc3223623 nodeName:}" failed. No retries permitted until 2026-10-30 06:17:09.997616612 +0000 UTC m=+1396.506636478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-maas-api-tls") pod "maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" (UID: "179d1033-3d70-4cca-a7cc-0a6bc3223623") : secret "maas-api-e2e-api-a-0fab6d-serving-cert" not found Oct 30 06:17:09.497881 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.497725 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9wt8\" (UniqueName: \"kubernetes.io/projected/179d1033-3d70-4cca-a7cc-0a6bc3223623-kube-api-access-z9wt8\") pod \"maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:09.498616 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.498589 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/179d1033-3d70-4cca-a7cc-0a6bc3223623-openshift-service-ca\") pod \"maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:09.500459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.500437 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-metrics-certs\") pod \"maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:09.521881 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.521854 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9wt8\" (UniqueName: \"kubernetes.io/projected/179d1033-3d70-4cca-a7cc-0a6bc3223623-kube-api-access-z9wt8\") pod \"maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:09.683341 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:09.683309 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:10.010033 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:10.009927 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-maas-api-tls\") pod \"maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:10.012619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:10.012593 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-maas-api-tls\") pod \"maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:10.214536 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:10.212529 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:10.551700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:10.551575 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk"] Oct 30 06:17:10.561025 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:10.561004 6999 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:17:10.671831 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:10.671789 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" event={"ID":"179d1033-3d70-4cca-a7cc-0a6bc3223623","Type":"ContainerStarted","Data":"19151ad007112f856acd7f82c012531a09b9cf32e5d60a89e56fa6b3cb1e9744"} Oct 30 06:17:11.386554 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:11.386518 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:11.642567 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:11.642503 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:17:11.679991 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:11.679964 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" event={"ID":"179d1033-3d70-4cca-a7cc-0a6bc3223623","Type":"ContainerStarted","Data":"1bb9df3e7157e437305d1062e1552b0db68c8618c1f5ccf682222d51fd2f3dbf"} Oct 30 06:17:11.680150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:11.680049 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:11.726968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:11.726920 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" podStartSLOduration=2.377875592 podStartE2EDuration="2.726905964s" totalImagesPullingTime="349.030372ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:11.726358337 +0000 UTC m=+1398.235378224" watchObservedRunningTime="2026-10-30 06:17:11.726905964 +0000 UTC m=+1398.235925849" Oct 30 06:17:13.335567 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:13.335524 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:13.509074 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:13.509041 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:15.057167 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:15.057108 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:15.212772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:15.212732 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:15.388069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:15.388042 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:17.021271 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:17.021243 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:17.692020 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:17.691993 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:17:18.856777 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:18.856747 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:20.756332 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:20.756301 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:22.014084 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:22.014055 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5"] Oct 30 06:17:22.171831 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:22.171792 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:22.340338 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:22.331668 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:22.548563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:22.548525 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:22.828329 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:22.828284 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:24.096995 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:24.096967 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82"] Oct 30 06:17:24.133471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:24.133441 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78"] Oct 30 06:17:24.286254 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:24.286224 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:24.440150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:24.440120 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:24.596352 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:24.596308 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:25.122231 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:25.122174 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:27.015032 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.014996 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" podUID="2d2f6047-f480-493a-9726-0e078ca0964e" containerName="payload-pre-processing" containerID="cri-o://671c4957a65be2ec8255282ea6f4a897d349f94ad663d21342acf84eaeb108fb" gracePeriod=25 Oct 30 06:17:27.019820 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.019799 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:27.321560 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.321530 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" Oct 30 06:17:27.466182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.466149 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2f6047-f480-493a-9726-0e078ca0964e-kube-api-access-xr8x5\" (UniqueName: \"kubernetes.io/projected/2d2f6047-f480-493a-9726-0e078ca0964e-kube-api-access-xr8x5\") pod \"2d2f6047-f480-493a-9726-0e078ca0964e\" (UID: \"2d2f6047-f480-493a-9726-0e078ca0964e\") " Oct 30 06:17:27.466391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.466365 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d2f6047-f480-493a-9726-0e078ca0964e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2f6047-f480-493a-9726-0e078ca0964e-plugins-config-volume\") pod \"2d2f6047-f480-493a-9726-0e078ca0964e\" (UID: \"2d2f6047-f480-493a-9726-0e078ca0964e\") " Oct 30 06:17:27.466711 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.466690 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2f6047-f480-493a-9726-0e078ca0964e-plugins-config-volume" pod "2d2f6047-f480-493a-9726-0e078ca0964e" (UID: "2d2f6047-f480-493a-9726-0e078ca0964e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:27.466927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.466910 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d2f6047-f480-493a-9726-0e078ca0964e-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:27.468270 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.468252 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2f6047-f480-493a-9726-0e078ca0964e-kube-api-access-xr8x5" pod "2d2f6047-f480-493a-9726-0e078ca0964e" (UID: "2d2f6047-f480-493a-9726-0e078ca0964e"). InnerVolumeSpecName "kube-api-access-xr8x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:27.568144 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.568081 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr8x5\" (UniqueName: \"kubernetes.io/projected/2d2f6047-f480-493a-9726-0e078ca0964e-kube-api-access-xr8x5\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:27.792517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.792487 6999 generic.go:356] "Generic (PLEG): container finished" podID="2d2f6047-f480-493a-9726-0e078ca0964e" containerID="671c4957a65be2ec8255282ea6f4a897d349f94ad663d21342acf84eaeb108fb" exitCode=0 Oct 30 06:17:27.792662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.792531 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" Oct 30 06:17:27.792662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.792571 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" event={"ID":"2d2f6047-f480-493a-9726-0e078ca0964e","Type":"ContainerDied","Data":"671c4957a65be2ec8255282ea6f4a897d349f94ad663d21342acf84eaeb108fb"} Oct 30 06:17:27.792662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.792623 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5" event={"ID":"2d2f6047-f480-493a-9726-0e078ca0964e","Type":"ContainerDied","Data":"f95838aac528ad97ae429c46cedfce2925f6bc4bb75e2077df1810f4fb17c461"} Oct 30 06:17:27.792662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.792651 6999 scope.go:122] "RemoveContainer" containerID="671c4957a65be2ec8255282ea6f4a897d349f94ad663d21342acf84eaeb108fb" Oct 30 06:17:27.802507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.802475 6999 scope.go:122] "RemoveContainer" containerID="671c4957a65be2ec8255282ea6f4a897d349f94ad663d21342acf84eaeb108fb" Oct 30 06:17:27.802790 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:17:27.802771 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"671c4957a65be2ec8255282ea6f4a897d349f94ad663d21342acf84eaeb108fb\": container with ID starting with 671c4957a65be2ec8255282ea6f4a897d349f94ad663d21342acf84eaeb108fb not found: ID does not exist" containerID="671c4957a65be2ec8255282ea6f4a897d349f94ad663d21342acf84eaeb108fb" Oct 30 06:17:27.802840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.802796 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671c4957a65be2ec8255282ea6f4a897d349f94ad663d21342acf84eaeb108fb"} err="failed to get container status \"671c4957a65be2ec8255282ea6f4a897d349f94ad663d21342acf84eaeb108fb\": rpc error: code = NotFound desc = could not find container \"671c4957a65be2ec8255282ea6f4a897d349f94ad663d21342acf84eaeb108fb\": container with ID starting with 671c4957a65be2ec8255282ea6f4a897d349f94ad663d21342acf84eaeb108fb not found: ID does not exist" Oct 30 06:17:27.847081 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.847023 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5"] Oct 30 06:17:27.858896 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:27.858863 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-7f645f5467-qjtc5"] Oct 30 06:17:28.843370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:28.843308 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:29.097524 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.097403 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" podUID="2e6de368-45ad-4356-8c54-66e69475abe2" containerName="payload-processing" containerID="cri-o://b98d983058e6caa16a0fe0f9d0defbf5f03915babe95a7eae4001f7c1b5edb6d" gracePeriod=25 Oct 30 06:17:29.134674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.134636 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" podUID="e85bdffe-82ea-4f28-870b-59983e2ce7df" containerName="payload-pre-processing" containerID="cri-o://941a023f9a8ec8c89a69a8fa5a5a4a2dd1162fc8d96af10f48923d71130ed0b0" gracePeriod=25 Oct 30 06:17:29.415576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.415552 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" Oct 30 06:17:29.443810 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.443787 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" Oct 30 06:17:29.599028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.598994 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e6de368-45ad-4356-8c54-66e69475abe2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e6de368-45ad-4356-8c54-66e69475abe2-plugins-config-volume\") pod \"2e6de368-45ad-4356-8c54-66e69475abe2\" (UID: \"2e6de368-45ad-4356-8c54-66e69475abe2\") " Oct 30 06:17:29.599199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.599110 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e6de368-45ad-4356-8c54-66e69475abe2-kube-api-access-9nzbx\" (UniqueName: \"kubernetes.io/projected/2e6de368-45ad-4356-8c54-66e69475abe2-kube-api-access-9nzbx\") pod \"2e6de368-45ad-4356-8c54-66e69475abe2\" (UID: \"2e6de368-45ad-4356-8c54-66e69475abe2\") " Oct 30 06:17:29.599199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.599141 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e85bdffe-82ea-4f28-870b-59983e2ce7df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e85bdffe-82ea-4f28-870b-59983e2ce7df-plugins-config-volume\") pod \"e85bdffe-82ea-4f28-870b-59983e2ce7df\" (UID: \"e85bdffe-82ea-4f28-870b-59983e2ce7df\") " Oct 30 06:17:29.599355 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.599343 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e85bdffe-82ea-4f28-870b-59983e2ce7df-kube-api-access-vbwvr\" (UniqueName: \"kubernetes.io/projected/e85bdffe-82ea-4f28-870b-59983e2ce7df-kube-api-access-vbwvr\") pod \"e85bdffe-82ea-4f28-870b-59983e2ce7df\" (UID: \"e85bdffe-82ea-4f28-870b-59983e2ce7df\") " Oct 30 06:17:29.599583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.599557 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6de368-45ad-4356-8c54-66e69475abe2-plugins-config-volume" pod "2e6de368-45ad-4356-8c54-66e69475abe2" (UID: "2e6de368-45ad-4356-8c54-66e69475abe2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:29.599661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.599632 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e85bdffe-82ea-4f28-870b-59983e2ce7df-plugins-config-volume" pod "e85bdffe-82ea-4f28-870b-59983e2ce7df" (UID: "e85bdffe-82ea-4f28-870b-59983e2ce7df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:29.600064 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.599971 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e85bdffe-82ea-4f28-870b-59983e2ce7df-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:29.600064 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.599991 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e6de368-45ad-4356-8c54-66e69475abe2-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:29.601456 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.601431 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6de368-45ad-4356-8c54-66e69475abe2-kube-api-access-9nzbx" pod "2e6de368-45ad-4356-8c54-66e69475abe2" (UID: "2e6de368-45ad-4356-8c54-66e69475abe2"). InnerVolumeSpecName "kube-api-access-9nzbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:29.601526 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.601508 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e85bdffe-82ea-4f28-870b-59983e2ce7df-kube-api-access-vbwvr" pod "e85bdffe-82ea-4f28-870b-59983e2ce7df" (UID: "e85bdffe-82ea-4f28-870b-59983e2ce7df"). InnerVolumeSpecName "kube-api-access-vbwvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:29.611780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.611756 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2f6047-f480-493a-9726-0e078ca0964e" path="/var/lib/kubelet/pods/2d2f6047-f480-493a-9726-0e078ca0964e/volumes" Oct 30 06:17:29.701346 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.701313 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbwvr\" (UniqueName: \"kubernetes.io/projected/e85bdffe-82ea-4f28-870b-59983e2ce7df-kube-api-access-vbwvr\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:29.701346 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.701343 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nzbx\" (UniqueName: \"kubernetes.io/projected/2e6de368-45ad-4356-8c54-66e69475abe2-kube-api-access-9nzbx\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:29.808843 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.808806 6999 generic.go:356] "Generic (PLEG): container finished" podID="e85bdffe-82ea-4f28-870b-59983e2ce7df" containerID="941a023f9a8ec8c89a69a8fa5a5a4a2dd1162fc8d96af10f48923d71130ed0b0" exitCode=0 Oct 30 06:17:29.809001 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.808865 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" Oct 30 06:17:29.809001 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.808887 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" event={"ID":"e85bdffe-82ea-4f28-870b-59983e2ce7df","Type":"ContainerDied","Data":"941a023f9a8ec8c89a69a8fa5a5a4a2dd1162fc8d96af10f48923d71130ed0b0"} Oct 30 06:17:29.809001 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.808975 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78" event={"ID":"e85bdffe-82ea-4f28-870b-59983e2ce7df","Type":"ContainerDied","Data":"3bdd121ecaffa98d4a9c4017f56c4a32e44e1a3ec6c1b47c71294d0367e94cf1"} Oct 30 06:17:29.809112 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.809007 6999 scope.go:122] "RemoveContainer" containerID="941a023f9a8ec8c89a69a8fa5a5a4a2dd1162fc8d96af10f48923d71130ed0b0" Oct 30 06:17:29.810416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.810394 6999 generic.go:356] "Generic (PLEG): container finished" podID="2e6de368-45ad-4356-8c54-66e69475abe2" containerID="b98d983058e6caa16a0fe0f9d0defbf5f03915babe95a7eae4001f7c1b5edb6d" exitCode=0 Oct 30 06:17:29.810521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.810446 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" Oct 30 06:17:29.810521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.810487 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" event={"ID":"2e6de368-45ad-4356-8c54-66e69475abe2","Type":"ContainerDied","Data":"b98d983058e6caa16a0fe0f9d0defbf5f03915babe95a7eae4001f7c1b5edb6d"} Oct 30 06:17:29.810637 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.810527 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82" event={"ID":"2e6de368-45ad-4356-8c54-66e69475abe2","Type":"ContainerDied","Data":"96e404388e1a61a93d543b22cd411ca05b4bf6b1c398455bdc16f80f414d85c9"} Oct 30 06:17:29.819824 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.819810 6999 scope.go:122] "RemoveContainer" containerID="941a023f9a8ec8c89a69a8fa5a5a4a2dd1162fc8d96af10f48923d71130ed0b0" Oct 30 06:17:29.820085 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:17:29.820065 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941a023f9a8ec8c89a69a8fa5a5a4a2dd1162fc8d96af10f48923d71130ed0b0\": container with ID starting with 941a023f9a8ec8c89a69a8fa5a5a4a2dd1162fc8d96af10f48923d71130ed0b0 not found: ID does not exist" containerID="941a023f9a8ec8c89a69a8fa5a5a4a2dd1162fc8d96af10f48923d71130ed0b0" Oct 30 06:17:29.820129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.820093 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941a023f9a8ec8c89a69a8fa5a5a4a2dd1162fc8d96af10f48923d71130ed0b0"} err="failed to get container status \"941a023f9a8ec8c89a69a8fa5a5a4a2dd1162fc8d96af10f48923d71130ed0b0\": rpc error: code = NotFound desc = could not find container \"941a023f9a8ec8c89a69a8fa5a5a4a2dd1162fc8d96af10f48923d71130ed0b0\": container with ID starting with 941a023f9a8ec8c89a69a8fa5a5a4a2dd1162fc8d96af10f48923d71130ed0b0 not found: ID does not exist" Oct 30 06:17:29.820129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.820108 6999 scope.go:122] "RemoveContainer" containerID="b98d983058e6caa16a0fe0f9d0defbf5f03915babe95a7eae4001f7c1b5edb6d" Oct 30 06:17:29.828883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.828869 6999 scope.go:122] "RemoveContainer" containerID="b98d983058e6caa16a0fe0f9d0defbf5f03915babe95a7eae4001f7c1b5edb6d" Oct 30 06:17:29.829130 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:17:29.829111 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98d983058e6caa16a0fe0f9d0defbf5f03915babe95a7eae4001f7c1b5edb6d\": container with ID starting with b98d983058e6caa16a0fe0f9d0defbf5f03915babe95a7eae4001f7c1b5edb6d not found: ID does not exist" containerID="b98d983058e6caa16a0fe0f9d0defbf5f03915babe95a7eae4001f7c1b5edb6d" Oct 30 06:17:29.829188 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.829138 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98d983058e6caa16a0fe0f9d0defbf5f03915babe95a7eae4001f7c1b5edb6d"} err="failed to get container status \"b98d983058e6caa16a0fe0f9d0defbf5f03915babe95a7eae4001f7c1b5edb6d\": rpc error: code = NotFound desc = could not find container \"b98d983058e6caa16a0fe0f9d0defbf5f03915babe95a7eae4001f7c1b5edb6d\": container with ID starting with b98d983058e6caa16a0fe0f9d0defbf5f03915babe95a7eae4001f7c1b5edb6d not found: ID does not exist" Oct 30 06:17:29.860706 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.860687 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82"] Oct 30 06:17:29.872342 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.872274 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584c9d-744b84fc6b-82z82"] Oct 30 06:17:29.897670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.897618 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78"] Oct 30 06:17:29.901893 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:29.901873 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-67d86d4c47-tvx78"] Oct 30 06:17:30.584344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.584301 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97"] Oct 30 06:17:30.584932 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.584915 6999 cpu_manager.go:414] "containerMap: removing container" podUID="2e6de368-45ad-4356-8c54-66e69475abe2" containerName="payload-processing" Oct 30 06:17:30.585041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.584934 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e6de368-45ad-4356-8c54-66e69475abe2" containerName="payload-processing" Oct 30 06:17:30.585041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.584965 6999 cpu_manager.go:414] "containerMap: removing container" podUID="2d2f6047-f480-493a-9726-0e078ca0964e" containerName="payload-pre-processing" Oct 30 06:17:30.585041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.584973 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2f6047-f480-493a-9726-0e078ca0964e" containerName="payload-pre-processing" Oct 30 06:17:30.585041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.585003 6999 cpu_manager.go:414] "containerMap: removing container" podUID="e85bdffe-82ea-4f28-870b-59983e2ce7df" containerName="payload-pre-processing" Oct 30 06:17:30.585041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.585011 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e85bdffe-82ea-4f28-870b-59983e2ce7df" containerName="payload-pre-processing" Oct 30 06:17:30.585317 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.585198 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e6de368-45ad-4356-8c54-66e69475abe2" containerName="payload-processing" Oct 30 06:17:30.585317 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.585231 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2f6047-f480-493a-9726-0e078ca0964e" containerName="payload-pre-processing" Oct 30 06:17:30.585317 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.585246 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="e85bdffe-82ea-4f28-870b-59983e2ce7df" containerName="payload-pre-processing" Oct 30 06:17:30.590203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.590184 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:30.607753 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.607733 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f50f6e-serving-cert\"" Oct 30 06:17:30.621070 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.621044 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97"] Oct 30 06:17:30.714089 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.714047 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c4bm\" (UniqueName: \"kubernetes.io/projected/e010dc1f-4cce-448d-bfea-0e499f8c03a4-kube-api-access-5c4bm\") pod \"maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:30.714338 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.714316 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e010dc1f-4cce-448d-bfea-0e499f8c03a4-metrics-certs\") pod \"maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:30.714499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.714477 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e010dc1f-4cce-448d-bfea-0e499f8c03a4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:30.714601 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.714581 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e010dc1f-4cce-448d-bfea-0e499f8c03a4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:30.782947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.782921 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:30.815595 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.815566 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e010dc1f-4cce-448d-bfea-0e499f8c03a4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:30.815777 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.815622 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e010dc1f-4cce-448d-bfea-0e499f8c03a4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:30.815777 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.815740 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c4bm\" (UniqueName: \"kubernetes.io/projected/e010dc1f-4cce-448d-bfea-0e499f8c03a4-kube-api-access-5c4bm\") pod \"maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:30.815905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.815838 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e010dc1f-4cce-448d-bfea-0e499f8c03a4-metrics-certs\") pod \"maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:30.816319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.816294 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e010dc1f-4cce-448d-bfea-0e499f8c03a4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:30.818654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.818632 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e010dc1f-4cce-448d-bfea-0e499f8c03a4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:30.818777 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.818683 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e010dc1f-4cce-448d-bfea-0e499f8c03a4-metrics-certs\") pod \"maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:30.839477 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.839409 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c4bm\" (UniqueName: \"kubernetes.io/projected/e010dc1f-4cce-448d-bfea-0e499f8c03a4-kube-api-access-5c4bm\") pod \"maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:30.903798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.903767 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:30.959550 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:30.959521 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:31.222095 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:31.222069 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97"] Oct 30 06:17:31.223070 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:17:31.223036 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode010dc1f_4cce_448d_bfea_0e499f8c03a4.slice/crio-8ef32512990fc70845e2c3621479c16135dcaba73b60f57b24bae01ce875ac2b WatchSource:0}: Error finding container 8ef32512990fc70845e2c3621479c16135dcaba73b60f57b24bae01ce875ac2b: Status 404 returned error can't find the container with id 8ef32512990fc70845e2c3621479c16135dcaba73b60f57b24bae01ce875ac2b Oct 30 06:17:31.614614 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:31.614582 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e6de368-45ad-4356-8c54-66e69475abe2" path="/var/lib/kubelet/pods/2e6de368-45ad-4356-8c54-66e69475abe2/volumes" Oct 30 06:17:31.615010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:31.614994 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e85bdffe-82ea-4f28-870b-59983e2ce7df" path="/var/lib/kubelet/pods/e85bdffe-82ea-4f28-870b-59983e2ce7df/volumes" Oct 30 06:17:31.826477 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:31.826438 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" event={"ID":"e010dc1f-4cce-448d-bfea-0e499f8c03a4","Type":"ContainerStarted","Data":"2139307acccc2484cccb6cd24304dc88cdaa7851e59e2c50fbb8d4177cb1a4a2"} Oct 30 06:17:31.826622 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:31.826500 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" event={"ID":"e010dc1f-4cce-448d-bfea-0e499f8c03a4","Type":"ContainerStarted","Data":"8ef32512990fc70845e2c3621479c16135dcaba73b60f57b24bae01ce875ac2b"} Oct 30 06:17:31.826622 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:31.826575 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:31.869959 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:31.869903 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" podStartSLOduration=1.511435526 podStartE2EDuration="1.86988992s" totalImagesPullingTime="358.454394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:31.86829284 +0000 UTC m=+1418.377312727" watchObservedRunningTime="2026-10-30 06:17:31.86988992 +0000 UTC m=+1418.378909805" Oct 30 06:17:32.357191 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.357104 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8"] Oct 30 06:17:32.357671 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.357386 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" podUID="4628f823-8667-4403-8fd5-2c27c8c76c23" containerName="maas-api" containerID="cri-o://10e1c137acdca01b43ed9a6593113224649f31e04c2c06676862672c5679b2d3" gracePeriod=30 Oct 30 06:17:32.466140 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.466105 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:32.754600 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.754347 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:17:32.758498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.758477 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-maas-api-tls\") pod \"4628f823-8667-4403-8fd5-2c27c8c76c23\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " Oct 30 06:17:32.758703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.758686 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-metrics-certs\") pod \"4628f823-8667-4403-8fd5-2c27c8c76c23\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " Oct 30 06:17:32.758877 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.758857 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4628f823-8667-4403-8fd5-2c27c8c76c23-kube-api-access-gfzvf\" (UniqueName: \"kubernetes.io/projected/4628f823-8667-4403-8fd5-2c27c8c76c23-kube-api-access-gfzvf\") pod \"4628f823-8667-4403-8fd5-2c27c8c76c23\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " Oct 30 06:17:32.759128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.759112 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4628f823-8667-4403-8fd5-2c27c8c76c23-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4628f823-8667-4403-8fd5-2c27c8c76c23-openshift-service-ca\") pod \"4628f823-8667-4403-8fd5-2c27c8c76c23\" (UID: \"4628f823-8667-4403-8fd5-2c27c8c76c23\") " Oct 30 06:17:32.760039 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.759547 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4628f823-8667-4403-8fd5-2c27c8c76c23-openshift-service-ca" pod "4628f823-8667-4403-8fd5-2c27c8c76c23" (UID: "4628f823-8667-4403-8fd5-2c27c8c76c23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:32.760039 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.759840 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4628f823-8667-4403-8fd5-2c27c8c76c23-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:32.761522 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.761491 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-metrics-certs" pod "4628f823-8667-4403-8fd5-2c27c8c76c23" (UID: "4628f823-8667-4403-8fd5-2c27c8c76c23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:17:32.762646 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.762622 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-maas-api-tls" pod "4628f823-8667-4403-8fd5-2c27c8c76c23" (UID: "4628f823-8667-4403-8fd5-2c27c8c76c23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:17:32.762728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.762654 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4628f823-8667-4403-8fd5-2c27c8c76c23-kube-api-access-gfzvf" pod "4628f823-8667-4403-8fd5-2c27c8c76c23" (UID: "4628f823-8667-4403-8fd5-2c27c8c76c23"). InnerVolumeSpecName "kube-api-access-gfzvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:32.841384 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.841336 6999 generic.go:356] "Generic (PLEG): container finished" podID="4628f823-8667-4403-8fd5-2c27c8c76c23" containerID="10e1c137acdca01b43ed9a6593113224649f31e04c2c06676862672c5679b2d3" exitCode=0 Oct 30 06:17:32.841553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.841389 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" Oct 30 06:17:32.841553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.841424 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" event={"ID":"4628f823-8667-4403-8fd5-2c27c8c76c23","Type":"ContainerDied","Data":"10e1c137acdca01b43ed9a6593113224649f31e04c2c06676862672c5679b2d3"} Oct 30 06:17:32.841553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.841498 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8" event={"ID":"4628f823-8667-4403-8fd5-2c27c8c76c23","Type":"ContainerDied","Data":"e6bea4bde7fd491e26675cb46acbc39aa413dedd07c44e126df872f70b84289c"} Oct 30 06:17:32.841553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.841538 6999 scope.go:122] "RemoveContainer" containerID="10e1c137acdca01b43ed9a6593113224649f31e04c2c06676862672c5679b2d3" Oct 30 06:17:32.857338 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.857208 6999 scope.go:122] "RemoveContainer" containerID="10e1c137acdca01b43ed9a6593113224649f31e04c2c06676862672c5679b2d3" Oct 30 06:17:32.857640 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:17:32.857615 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e1c137acdca01b43ed9a6593113224649f31e04c2c06676862672c5679b2d3\": container with ID starting with 10e1c137acdca01b43ed9a6593113224649f31e04c2c06676862672c5679b2d3 not found: ID does not exist" containerID="10e1c137acdca01b43ed9a6593113224649f31e04c2c06676862672c5679b2d3" Oct 30 06:17:32.857723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.857650 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e1c137acdca01b43ed9a6593113224649f31e04c2c06676862672c5679b2d3"} err="failed to get container status \"10e1c137acdca01b43ed9a6593113224649f31e04c2c06676862672c5679b2d3\": rpc error: code = NotFound desc = could not find container \"10e1c137acdca01b43ed9a6593113224649f31e04c2c06676862672c5679b2d3\": container with ID starting with 10e1c137acdca01b43ed9a6593113224649f31e04c2c06676862672c5679b2d3 not found: ID does not exist" Oct 30 06:17:32.862062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.862038 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:32.862143 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.862066 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4628f823-8667-4403-8fd5-2c27c8c76c23-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:32.862143 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.862076 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfzvf\" (UniqueName: \"kubernetes.io/projected/4628f823-8667-4403-8fd5-2c27c8c76c23-kube-api-access-gfzvf\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:32.897164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.897135 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8"] Oct 30 06:17:32.906874 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:32.906838 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7a606-59df8df945-qskw8"] Oct 30 06:17:33.045297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:33.044981 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:33.218535 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:33.218502 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:33.403200 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:33.403163 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:33.609655 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:33.609626 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4628f823-8667-4403-8fd5-2c27c8c76c23" path="/var/lib/kubelet/pods/4628f823-8667-4403-8fd5-2c27c8c76c23/volumes" Oct 30 06:17:33.736401 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:33.736307 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:34.141356 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:17:34.141319 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:17:34.465527 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:34.465449 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:35.180201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:35.178918 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:35.345990 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:35.345962 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:35.669036 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:35.668993 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc"] Oct 30 06:17:35.672435 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:35.672388 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" podUID="0fa2534f-7589-4a20-b321-10df418b86c5" containerName="istio-proxy" containerID="cri-o://b612bbc37ba6b8f11cff71b7f827f7c230a12b9ef80c99d71a1c188247de47b4" gracePeriod=30 Oct 30 06:17:35.691013 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:35.687467 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs"] Oct 30 06:17:35.691013 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:35.687718 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" podUID="43830d5c-1f9d-4699-a0f8-089f40e1e098" containerName="istio-proxy" containerID="cri-o://89ea3ecaf3be0239493b96241d9ab807609fd98ee42d1d3a674d293680d6cf04" gracePeriod=30 Oct 30 06:17:35.696990 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:35.696963 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89"] Oct 30 06:17:35.697264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:35.697175 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" podUID="bbd16456-e1e8-4295-8d4a-3045f9e20db1" containerName="istio-proxy" containerID="cri-o://57cbd05bb0588fb3cfe0b0575a8adbc6e9e549533e0a7f0f8956d76e67553dc7" gracePeriod=30 Oct 30 06:17:35.701413 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:35.700821 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7"] Oct 30 06:17:35.701413 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:35.700993 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" podUID="6109420f-b8c7-4f71-8796-5e79cdcd53b0" containerName="istio-proxy" containerID="cri-o://ea28ad044f2c7e24f53b338f5049b871be3017b209c0767c5967f47f392465b5" gracePeriod=30 Oct 30 06:17:35.776682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:35.776642 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:35.945296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:35.943364 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:37.481745 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:37.481709 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:37.846573 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:37.846498 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:17:39.473837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.473804 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d"] Oct 30 06:17:39.474480 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.474464 6999 cpu_manager.go:414] "containerMap: removing container" podUID="4628f823-8667-4403-8fd5-2c27c8c76c23" containerName="maas-api" Oct 30 06:17:39.474537 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.474481 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4628f823-8667-4403-8fd5-2c27c8c76c23" containerName="maas-api" Oct 30 06:17:39.474596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.474587 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="4628f823-8667-4403-8fd5-2c27c8c76c23" containerName="maas-api" Oct 30 06:17:39.478361 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.478331 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" Oct 30 06:17:39.499297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.499267 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d"] Oct 30 06:17:39.507698 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.507672 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-33c99c\"" Oct 30 06:17:39.507854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.507707 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-33c99c-dockercfg-tmvtv\"" Oct 30 06:17:39.558038 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.558005 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf"] Oct 30 06:17:39.562631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.562612 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" Oct 30 06:17:39.570807 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.570777 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f277c85-c9a4-47ed-9c18-96fcdf78728e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d\" (UID: \"8f277c85-c9a4-47ed-9c18-96fcdf78728e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" Oct 30 06:17:39.571404 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.571377 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqvmj\" (UniqueName: \"kubernetes.io/projected/8f277c85-c9a4-47ed-9c18-96fcdf78728e-kube-api-access-hqvmj\") pod \"payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d\" (UID: \"8f277c85-c9a4-47ed-9c18-96fcdf78728e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" Oct 30 06:17:39.593027 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.592964 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf"] Oct 30 06:17:39.672694 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.672660 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62893d3-85ac-42d9-a5f4-e7eba0aa5924-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf\" (UID: \"c62893d3-85ac-42d9-a5f4-e7eba0aa5924\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" Oct 30 06:17:39.672969 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.672949 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f277c85-c9a4-47ed-9c18-96fcdf78728e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d\" (UID: \"8f277c85-c9a4-47ed-9c18-96fcdf78728e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" Oct 30 06:17:39.673064 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.672987 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzpxc\" (UniqueName: \"kubernetes.io/projected/c62893d3-85ac-42d9-a5f4-e7eba0aa5924-kube-api-access-fzpxc\") pod \"payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf\" (UID: \"c62893d3-85ac-42d9-a5f4-e7eba0aa5924\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" Oct 30 06:17:39.673064 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.673046 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqvmj\" (UniqueName: \"kubernetes.io/projected/8f277c85-c9a4-47ed-9c18-96fcdf78728e-kube-api-access-hqvmj\") pod \"payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d\" (UID: \"8f277c85-c9a4-47ed-9c18-96fcdf78728e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" Oct 30 06:17:39.673745 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.673723 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f277c85-c9a4-47ed-9c18-96fcdf78728e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d\" (UID: \"8f277c85-c9a4-47ed-9c18-96fcdf78728e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" Oct 30 06:17:39.697840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.697805 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqvmj\" (UniqueName: \"kubernetes.io/projected/8f277c85-c9a4-47ed-9c18-96fcdf78728e-kube-api-access-hqvmj\") pod \"payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d\" (UID: \"8f277c85-c9a4-47ed-9c18-96fcdf78728e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" Oct 30 06:17:39.775108 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.775019 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62893d3-85ac-42d9-a5f4-e7eba0aa5924-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf\" (UID: \"c62893d3-85ac-42d9-a5f4-e7eba0aa5924\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" Oct 30 06:17:39.775320 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.775236 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzpxc\" (UniqueName: \"kubernetes.io/projected/c62893d3-85ac-42d9-a5f4-e7eba0aa5924-kube-api-access-fzpxc\") pod \"payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf\" (UID: \"c62893d3-85ac-42d9-a5f4-e7eba0aa5924\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" Oct 30 06:17:39.775869 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.775841 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62893d3-85ac-42d9-a5f4-e7eba0aa5924-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf\" (UID: \"c62893d3-85ac-42d9-a5f4-e7eba0aa5924\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" Oct 30 06:17:39.795620 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.795588 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" Oct 30 06:17:39.799773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.799739 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzpxc\" (UniqueName: \"kubernetes.io/projected/c62893d3-85ac-42d9-a5f4-e7eba0aa5924-kube-api-access-fzpxc\") pod \"payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf\" (UID: \"c62893d3-85ac-42d9-a5f4-e7eba0aa5924\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" Oct 30 06:17:39.899856 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:39.897889 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" Oct 30 06:17:40.126657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:40.126629 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d"] Oct 30 06:17:40.127743 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:17:40.127718 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f277c85_c9a4_47ed_9c18_96fcdf78728e.slice/crio-233bbf3814c689e5d4170f8b576112d0afbf541cc6abcaf3601b9971bf8bccd7 WatchSource:0}: Error finding container 233bbf3814c689e5d4170f8b576112d0afbf541cc6abcaf3601b9971bf8bccd7: Status 404 returned error can't find the container with id 233bbf3814c689e5d4170f8b576112d0afbf541cc6abcaf3601b9971bf8bccd7 Oct 30 06:17:40.250235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:40.250196 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf"] Oct 30 06:17:40.251842 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:17:40.251816 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62893d3_85ac_42d9_a5f4_e7eba0aa5924.slice/crio-47306fefeaa991f8307f5018a9832f0ca5d9987b19f0a169853abadadf0b51dd WatchSource:0}: Error finding container 47306fefeaa991f8307f5018a9832f0ca5d9987b19f0a169853abadadf0b51dd: Status 404 returned error can't find the container with id 47306fefeaa991f8307f5018a9832f0ca5d9987b19f0a169853abadadf0b51dd Oct 30 06:17:40.479623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:40.479526 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:40.631338 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:40.631308 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:40.957890 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:40.957859 6999 generic.go:356] "Generic (PLEG): container finished" podID="0fa2534f-7589-4a20-b321-10df418b86c5" containerID="b612bbc37ba6b8f11cff71b7f827f7c230a12b9ef80c99d71a1c188247de47b4" exitCode=0 Oct 30 06:17:40.959237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:40.958065 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" event={"ID":"0fa2534f-7589-4a20-b321-10df418b86c5","Type":"ContainerDied","Data":"b612bbc37ba6b8f11cff71b7f827f7c230a12b9ef80c99d71a1c188247de47b4"} Oct 30 06:17:40.973285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:40.960099 6999 generic.go:356] "Generic (PLEG): container finished" podID="6109420f-b8c7-4f71-8796-5e79cdcd53b0" containerID="ea28ad044f2c7e24f53b338f5049b871be3017b209c0767c5967f47f392465b5" exitCode=0 Oct 30 06:17:40.973285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:40.960182 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" event={"ID":"6109420f-b8c7-4f71-8796-5e79cdcd53b0","Type":"ContainerDied","Data":"ea28ad044f2c7e24f53b338f5049b871be3017b209c0767c5967f47f392465b5"} Oct 30 06:17:40.973285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:40.963731 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" event={"ID":"c62893d3-85ac-42d9-a5f4-e7eba0aa5924","Type":"ContainerStarted","Data":"47306fefeaa991f8307f5018a9832f0ca5d9987b19f0a169853abadadf0b51dd"} Oct 30 06:17:40.973285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:40.966916 6999 generic.go:356] "Generic (PLEG): container finished" podID="bbd16456-e1e8-4295-8d4a-3045f9e20db1" containerID="57cbd05bb0588fb3cfe0b0575a8adbc6e9e549533e0a7f0f8956d76e67553dc7" exitCode=0 Oct 30 06:17:40.973285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:40.966992 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" event={"ID":"bbd16456-e1e8-4295-8d4a-3045f9e20db1","Type":"ContainerDied","Data":"57cbd05bb0588fb3cfe0b0575a8adbc6e9e549533e0a7f0f8956d76e67553dc7"} Oct 30 06:17:40.973285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:40.967037 6999 scope.go:122] "RemoveContainer" containerID="56a7548fb3a375198e80cf48bebce5cecb927a3f536d32933e87ef09775b8f06" Oct 30 06:17:40.973285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:40.970979 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" event={"ID":"8f277c85-c9a4-47ed-9c18-96fcdf78728e","Type":"ContainerStarted","Data":"233bbf3814c689e5d4170f8b576112d0afbf541cc6abcaf3601b9971bf8bccd7"} Oct 30 06:17:40.973285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:40.973239 6999 generic.go:356] "Generic (PLEG): container finished" podID="43830d5c-1f9d-4699-a0f8-089f40e1e098" containerID="89ea3ecaf3be0239493b96241d9ab807609fd98ee42d1d3a674d293680d6cf04" exitCode=0 Oct 30 06:17:40.973859 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:40.973307 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" event={"ID":"43830d5c-1f9d-4699-a0f8-089f40e1e098","Type":"ContainerDied","Data":"89ea3ecaf3be0239493b96241d9ab807609fd98ee42d1d3a674d293680d6cf04"} Oct 30 06:17:41.033509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.033444 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:41.043557 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.043515 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:17:41.114250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.114153 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6109420f-b8c7-4f71-8796-5e79cdcd53b0-kube-api-access-8frvh\" (UniqueName: \"kubernetes.io/projected/6109420f-b8c7-4f71-8796-5e79cdcd53b0-kube-api-access-8frvh\") pod \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " Oct 30 06:17:41.114387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.114266 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-token\" (UniqueName: \"kubernetes.io/projected/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-token\") pod \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " Oct 30 06:17:41.114455 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.114430 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-workload-socket\") pod \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " Oct 30 06:17:41.114563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.114546 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-workload-certs\") pod \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " Oct 30 06:17:41.114682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.114665 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-envoy\") pod \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " Oct 30 06:17:41.114764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.114729 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-data\") pod \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " Oct 30 06:17:41.114860 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.114791 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-podinfo\") pod \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " Oct 30 06:17:41.114860 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.114837 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-credential-socket\") pod \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " Oct 30 06:17:41.114961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.114910 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istiod-ca-cert\") pod \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\" (UID: \"6109420f-b8c7-4f71-8796-5e79cdcd53b0\") " Oct 30 06:17:41.116424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.115139 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-workload-socket" pod "6109420f-b8c7-4f71-8796-5e79cdcd53b0" (UID: "6109420f-b8c7-4f71-8796-5e79cdcd53b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.116424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.115255 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-data" pod "6109420f-b8c7-4f71-8796-5e79cdcd53b0" (UID: "6109420f-b8c7-4f71-8796-5e79cdcd53b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.116424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.115430 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-workload-certs" pod "6109420f-b8c7-4f71-8796-5e79cdcd53b0" (UID: "6109420f-b8c7-4f71-8796-5e79cdcd53b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.116424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.115523 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istiod-ca-cert" pod "6109420f-b8c7-4f71-8796-5e79cdcd53b0" (UID: "6109420f-b8c7-4f71-8796-5e79cdcd53b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:41.116424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.115526 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-credential-socket" pod "6109420f-b8c7-4f71-8796-5e79cdcd53b0" (UID: "6109420f-b8c7-4f71-8796-5e79cdcd53b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.116424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.115737 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.116424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.115761 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.116424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.115779 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.116424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.115792 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.116424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.115804 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.120263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.120224 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-podinfo" pod "6109420f-b8c7-4f71-8796-5e79cdcd53b0" (UID: "6109420f-b8c7-4f71-8796-5e79cdcd53b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:41.123160 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.123133 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6109420f-b8c7-4f71-8796-5e79cdcd53b0-kube-api-access-8frvh" pod "6109420f-b8c7-4f71-8796-5e79cdcd53b0" (UID: "6109420f-b8c7-4f71-8796-5e79cdcd53b0"). InnerVolumeSpecName "kube-api-access-8frvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:41.129583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.129190 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:17:41.135506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.135447 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-token" pod "6109420f-b8c7-4f71-8796-5e79cdcd53b0" (UID: "6109420f-b8c7-4f71-8796-5e79cdcd53b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:41.137148 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.137107 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-envoy" pod "6109420f-b8c7-4f71-8796-5e79cdcd53b0" (UID: "6109420f-b8c7-4f71-8796-5e79cdcd53b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.139147 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.139068 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:17:41.169723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.169696 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:17:41.217452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217044 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-token\" (UniqueName: \"kubernetes.io/projected/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-token\") pod \"43830d5c-1f9d-4699-a0f8-089f40e1e098\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " Oct 30 06:17:41.217452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217147 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-workload-socket\") pod \"0fa2534f-7589-4a20-b321-10df418b86c5\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " Oct 30 06:17:41.217452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217225 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa2534f-7589-4a20-b321-10df418b86c5-istio-token\" (UniqueName: \"kubernetes.io/projected/0fa2534f-7589-4a20-b321-10df418b86c5-istio-token\") pod \"0fa2534f-7589-4a20-b321-10df418b86c5\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " Oct 30 06:17:41.217452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217274 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-data\") pod \"43830d5c-1f9d-4699-a0f8-089f40e1e098\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " Oct 30 06:17:41.217452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217307 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa2534f-7589-4a20-b321-10df418b86c5-kube-api-access-bnjj2\" (UniqueName: \"kubernetes.io/projected/0fa2534f-7589-4a20-b321-10df418b86c5-kube-api-access-bnjj2\") pod \"0fa2534f-7589-4a20-b321-10df418b86c5\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " Oct 30 06:17:41.217452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217331 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-workload-certs\") pod \"0fa2534f-7589-4a20-b321-10df418b86c5\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " Oct 30 06:17:41.217452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217363 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fa2534f-7589-4a20-b321-10df418b86c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fa2534f-7589-4a20-b321-10df418b86c5-istio-podinfo\") pod \"0fa2534f-7589-4a20-b321-10df418b86c5\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " Oct 30 06:17:41.217452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217389 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-workload-socket\") pod \"43830d5c-1f9d-4699-a0f8-089f40e1e098\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " Oct 30 06:17:41.217452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217421 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-workload-certs\") pod \"43830d5c-1f9d-4699-a0f8-089f40e1e098\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " Oct 30 06:17:41.217452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217469 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-podinfo\") pod \"43830d5c-1f9d-4699-a0f8-089f40e1e098\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " Oct 30 06:17:41.218374 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217572 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-envoy\") pod \"43830d5c-1f9d-4699-a0f8-089f40e1e098\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " Oct 30 06:17:41.218374 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217627 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-credential-socket\") pod \"43830d5c-1f9d-4699-a0f8-089f40e1e098\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " Oct 30 06:17:41.218374 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217657 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-istio-data\") pod \"0fa2534f-7589-4a20-b321-10df418b86c5\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " Oct 30 06:17:41.218374 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217712 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-data" pod "43830d5c-1f9d-4699-a0f8-089f40e1e098" (UID: "43830d5c-1f9d-4699-a0f8-089f40e1e098"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.218374 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217717 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43830d5c-1f9d-4699-a0f8-089f40e1e098-kube-api-access-2b54x\" (UniqueName: \"kubernetes.io/projected/43830d5c-1f9d-4699-a0f8-089f40e1e098-kube-api-access-2b54x\") pod \"43830d5c-1f9d-4699-a0f8-089f40e1e098\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " Oct 30 06:17:41.218374 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.217796 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa2534f-7589-4a20-b321-10df418b86c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fa2534f-7589-4a20-b321-10df418b86c5-istiod-ca-cert\") pod \"0fa2534f-7589-4a20-b321-10df418b86c5\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " Oct 30 06:17:41.221063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.218781 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-workload-certs" pod "43830d5c-1f9d-4699-a0f8-089f40e1e098" (UID: "43830d5c-1f9d-4699-a0f8-089f40e1e098"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.221063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.219792 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-workload-certs" pod "0fa2534f-7589-4a20-b321-10df418b86c5" (UID: "0fa2534f-7589-4a20-b321-10df418b86c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.221063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.220437 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-istio-data" pod "0fa2534f-7589-4a20-b321-10df418b86c5" (UID: "0fa2534f-7589-4a20-b321-10df418b86c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.221063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.220499 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-workload-socket" pod "0fa2534f-7589-4a20-b321-10df418b86c5" (UID: "0fa2534f-7589-4a20-b321-10df418b86c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.221063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.220737 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-workload-socket" pod "43830d5c-1f9d-4699-a0f8-089f40e1e098" (UID: "43830d5c-1f9d-4699-a0f8-089f40e1e098"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.221063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.220985 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa2534f-7589-4a20-b321-10df418b86c5-istiod-ca-cert" pod "0fa2534f-7589-4a20-b321-10df418b86c5" (UID: "0fa2534f-7589-4a20-b321-10df418b86c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:41.221063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.221040 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-credential-socket" pod "43830d5c-1f9d-4699-a0f8-089f40e1e098" (UID: "43830d5c-1f9d-4699-a0f8-089f40e1e098"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.221736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.221705 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-podinfo" pod "43830d5c-1f9d-4699-a0f8-089f40e1e098" (UID: "43830d5c-1f9d-4699-a0f8-089f40e1e098"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:41.221863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.221807 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43830d5c-1f9d-4699-a0f8-089f40e1e098-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43830d5c-1f9d-4699-a0f8-089f40e1e098-istiod-ca-cert\") pod \"43830d5c-1f9d-4699-a0f8-089f40e1e098\" (UID: \"43830d5c-1f9d-4699-a0f8-089f40e1e098\") " Oct 30 06:17:41.222328 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.222294 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43830d5c-1f9d-4699-a0f8-089f40e1e098-istiod-ca-cert" pod "43830d5c-1f9d-4699-a0f8-089f40e1e098" (UID: "43830d5c-1f9d-4699-a0f8-089f40e1e098"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:41.222488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.222466 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-istio-envoy\") pod \"0fa2534f-7589-4a20-b321-10df418b86c5\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " Oct 30 06:17:41.222633 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.222525 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-credential-socket\") pod \"0fa2534f-7589-4a20-b321-10df418b86c5\" (UID: \"0fa2534f-7589-4a20-b321-10df418b86c5\") " Oct 30 06:17:41.222850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.222830 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-credential-socket" pod "0fa2534f-7589-4a20-b321-10df418b86c5" (UID: "0fa2534f-7589-4a20-b321-10df418b86c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.228555 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.228521 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-istio-envoy" pod "0fa2534f-7589-4a20-b321-10df418b86c5" (UID: "0fa2534f-7589-4a20-b321-10df418b86c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.229932 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.229952 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.229963 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.229977 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.229986 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.229994 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.230002 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.230010 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.230017 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fa2534f-7589-4a20-b321-10df418b86c5-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.230025 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.230031 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43830d5c-1f9d-4699-a0f8-089f40e1e098-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.230041 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.230053 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.230061 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8frvh\" (UniqueName: \"kubernetes.io/projected/6109420f-b8c7-4f71-8796-5e79cdcd53b0-kube-api-access-8frvh\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.230070 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fa2534f-7589-4a20-b321-10df418b86c5-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.230078 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6109420f-b8c7-4f71-8796-5e79cdcd53b0-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.233093 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-envoy" pod "43830d5c-1f9d-4699-a0f8-089f40e1e098" (UID: "43830d5c-1f9d-4699-a0f8-089f40e1e098"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.235780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.233830 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-token" pod "43830d5c-1f9d-4699-a0f8-089f40e1e098" (UID: "43830d5c-1f9d-4699-a0f8-089f40e1e098"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:41.237007 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.236603 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:41.241655 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.241556 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fa2534f-7589-4a20-b321-10df418b86c5-istio-podinfo" pod "0fa2534f-7589-4a20-b321-10df418b86c5" (UID: "0fa2534f-7589-4a20-b321-10df418b86c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:41.241825 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.241741 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa2534f-7589-4a20-b321-10df418b86c5-istio-token" pod "0fa2534f-7589-4a20-b321-10df418b86c5" (UID: "0fa2534f-7589-4a20-b321-10df418b86c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:41.243298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.243257 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43830d5c-1f9d-4699-a0f8-089f40e1e098-kube-api-access-2b54x" pod "43830d5c-1f9d-4699-a0f8-089f40e1e098" (UID: "43830d5c-1f9d-4699-a0f8-089f40e1e098"). InnerVolumeSpecName "kube-api-access-2b54x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:41.244073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.244048 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa2534f-7589-4a20-b321-10df418b86c5-kube-api-access-bnjj2" pod "0fa2534f-7589-4a20-b321-10df418b86c5" (UID: "0fa2534f-7589-4a20-b321-10df418b86c5"). InnerVolumeSpecName "kube-api-access-bnjj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:41.331612 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.331524 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-workload-socket\") pod \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " Oct 30 06:17:41.331612 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.331570 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-podinfo\") pod \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " Oct 30 06:17:41.331612 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.331605 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-envoy\") pod \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " Oct 30 06:17:41.331916 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.331812 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-workload-socket" pod "bbd16456-e1e8-4295-8d4a-3045f9e20db1" (UID: "bbd16456-e1e8-4295-8d4a-3045f9e20db1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.332336 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.332311 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-workload-certs\") pod \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " Oct 30 06:17:41.332481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.332432 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-credential-socket\") pod \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " Oct 30 06:17:41.332481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.332473 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd16456-e1e8-4295-8d4a-3045f9e20db1-kube-api-access-cvtr8\" (UniqueName: \"kubernetes.io/projected/bbd16456-e1e8-4295-8d4a-3045f9e20db1-kube-api-access-cvtr8\") pod \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " Oct 30 06:17:41.332608 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.332509 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-token\" (UniqueName: \"kubernetes.io/projected/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-token\") pod \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " Oct 30 06:17:41.332608 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.332547 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-data\") pod \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " Oct 30 06:17:41.332787 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.332663 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-workload-certs" pod "bbd16456-e1e8-4295-8d4a-3045f9e20db1" (UID: "bbd16456-e1e8-4295-8d4a-3045f9e20db1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.332787 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.332719 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istiod-ca-cert\") pod \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\" (UID: \"bbd16456-e1e8-4295-8d4a-3045f9e20db1\") " Oct 30 06:17:41.332980 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.332954 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-data" pod "bbd16456-e1e8-4295-8d4a-3045f9e20db1" (UID: "bbd16456-e1e8-4295-8d4a-3045f9e20db1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.333114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.333093 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-credential-socket" pod "bbd16456-e1e8-4295-8d4a-3045f9e20db1" (UID: "bbd16456-e1e8-4295-8d4a-3045f9e20db1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.333756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.333589 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.333756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.333641 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fa2534f-7589-4a20-b321-10df418b86c5-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.333756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.333654 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.333756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.333669 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnjj2\" (UniqueName: \"kubernetes.io/projected/0fa2534f-7589-4a20-b321-10df418b86c5-kube-api-access-bnjj2\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.333756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.333685 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.333756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.333698 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fa2534f-7589-4a20-b321-10df418b86c5-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.333756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.333711 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.333756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.333727 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43830d5c-1f9d-4699-a0f8-089f40e1e098-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.333756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.333739 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.333756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.333751 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b54x\" (UniqueName: \"kubernetes.io/projected/43830d5c-1f9d-4699-a0f8-089f40e1e098-kube-api-access-2b54x\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.334498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.333913 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istiod-ca-cert" pod "bbd16456-e1e8-4295-8d4a-3045f9e20db1" (UID: "bbd16456-e1e8-4295-8d4a-3045f9e20db1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:41.335282 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.335256 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-envoy" pod "bbd16456-e1e8-4295-8d4a-3045f9e20db1" (UID: "bbd16456-e1e8-4295-8d4a-3045f9e20db1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:41.335683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.335661 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd16456-e1e8-4295-8d4a-3045f9e20db1-kube-api-access-cvtr8" pod "bbd16456-e1e8-4295-8d4a-3045f9e20db1" (UID: "bbd16456-e1e8-4295-8d4a-3045f9e20db1"). InnerVolumeSpecName "kube-api-access-cvtr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:41.335830 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.335782 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-token" pod "bbd16456-e1e8-4295-8d4a-3045f9e20db1" (UID: "bbd16456-e1e8-4295-8d4a-3045f9e20db1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:41.338741 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.338699 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-podinfo" pod "bbd16456-e1e8-4295-8d4a-3045f9e20db1" (UID: "bbd16456-e1e8-4295-8d4a-3045f9e20db1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:41.416404 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.416366 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:41.435728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.435696 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvtr8\" (UniqueName: \"kubernetes.io/projected/bbd16456-e1e8-4295-8d4a-3045f9e20db1-kube-api-access-cvtr8\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.435728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.435727 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.435936 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.435742 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.435936 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.435755 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.435936 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.435769 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbd16456-e1e8-4295-8d4a-3045f9e20db1-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:17:41.576107 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.576017 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:41.851519 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.851485 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:41.985862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.985075 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" Oct 30 06:17:41.985862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.985101 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc" event={"ID":"0fa2534f-7589-4a20-b321-10df418b86c5","Type":"ContainerDied","Data":"ab62d839f66ee379cb0952002e4c5e56c343cd3b13c05afd49b0a9396a13e607"} Oct 30 06:17:41.985862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.985172 6999 scope.go:122] "RemoveContainer" containerID="b612bbc37ba6b8f11cff71b7f827f7c230a12b9ef80c99d71a1c188247de47b4" Oct 30 06:17:41.987903 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.987330 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" Oct 30 06:17:41.987903 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.987605 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7" event={"ID":"6109420f-b8c7-4f71-8796-5e79cdcd53b0","Type":"ContainerDied","Data":"7698a77ca02d2e4d45e809256cfe662461af28f5b22a0d47fe69ffce2034308c"} Oct 30 06:17:41.990323 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.989665 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" event={"ID":"bbd16456-e1e8-4295-8d4a-3045f9e20db1","Type":"ContainerDied","Data":"8a1e66d7e0628ac950d0f0ba2885cb6886a354bd9e2e585bfbc5cfd2a29c338b"} Oct 30 06:17:41.990323 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.989761 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89" Oct 30 06:17:41.993534 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.992892 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" event={"ID":"43830d5c-1f9d-4699-a0f8-089f40e1e098","Type":"ContainerDied","Data":"2bc172a126c7a9f2e756f66ea6549448c80f0de0f64663df32b6974af65f04b1"} Oct 30 06:17:41.993534 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:41.992983 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs" Oct 30 06:17:42.001733 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.001709 6999 scope.go:122] "RemoveContainer" containerID="ea28ad044f2c7e24f53b338f5049b871be3017b209c0767c5967f47f392465b5" Oct 30 06:17:42.015914 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.015823 6999 scope.go:122] "RemoveContainer" containerID="57cbd05bb0588fb3cfe0b0575a8adbc6e9e549533e0a7f0f8956d76e67553dc7" Oct 30 06:17:42.042309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.039181 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89"] Oct 30 06:17:42.054645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.054617 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-thg89"] Oct 30 06:17:42.068901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.068874 6999 scope.go:122] "RemoveContainer" containerID="89ea3ecaf3be0239493b96241d9ab807609fd98ee42d1d3a674d293680d6cf04" Oct 30 06:17:42.078968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.078941 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc"] Oct 30 06:17:42.082349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.082253 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-nr5zc"] Oct 30 06:17:42.121700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.121639 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7"] Oct 30 06:17:42.125330 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.125303 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-mjcw7"] Oct 30 06:17:42.160987 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.157966 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs"] Oct 30 06:17:42.168883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.168854 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9hmcs"] Oct 30 06:17:42.515166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.515090 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:42.782413 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.782387 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:42.943985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.943955 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd"] Oct 30 06:17:42.944545 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.944532 6999 cpu_manager.go:414] "containerMap: removing container" podUID="bbd16456-e1e8-4295-8d4a-3045f9e20db1" containerName="istio-proxy" Oct 30 06:17:42.944597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.944547 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd16456-e1e8-4295-8d4a-3045f9e20db1" containerName="istio-proxy" Oct 30 06:17:42.944597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.944560 6999 cpu_manager.go:414] "containerMap: removing container" podUID="6109420f-b8c7-4f71-8796-5e79cdcd53b0" containerName="istio-proxy" Oct 30 06:17:42.944597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.944564 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6109420f-b8c7-4f71-8796-5e79cdcd53b0" containerName="istio-proxy" Oct 30 06:17:42.944597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.944572 6999 cpu_manager.go:414] "containerMap: removing container" podUID="bbd16456-e1e8-4295-8d4a-3045f9e20db1" containerName="istio-proxy" Oct 30 06:17:42.944597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.944576 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd16456-e1e8-4295-8d4a-3045f9e20db1" containerName="istio-proxy" Oct 30 06:17:42.944597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.944590 6999 cpu_manager.go:414] "containerMap: removing container" podUID="0fa2534f-7589-4a20-b321-10df418b86c5" containerName="istio-proxy" Oct 30 06:17:42.944597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.944594 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa2534f-7589-4a20-b321-10df418b86c5" containerName="istio-proxy" Oct 30 06:17:42.944819 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.944608 6999 cpu_manager.go:414] "containerMap: removing container" podUID="43830d5c-1f9d-4699-a0f8-089f40e1e098" containerName="istio-proxy" Oct 30 06:17:42.944819 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.944612 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43830d5c-1f9d-4699-a0f8-089f40e1e098" containerName="istio-proxy" Oct 30 06:17:42.945566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.945543 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="43830d5c-1f9d-4699-a0f8-089f40e1e098" containerName="istio-proxy" Oct 30 06:17:42.945673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.945601 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd16456-e1e8-4295-8d4a-3045f9e20db1" containerName="istio-proxy" Oct 30 06:17:42.945673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.945613 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="6109420f-b8c7-4f71-8796-5e79cdcd53b0" containerName="istio-proxy" Oct 30 06:17:42.945673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.945631 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd16456-e1e8-4295-8d4a-3045f9e20db1" containerName="istio-proxy" Oct 30 06:17:42.945673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.945648 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa2534f-7589-4a20-b321-10df418b86c5" containerName="istio-proxy" Oct 30 06:17:42.950779 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.950755 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" Oct 30 06:17:42.957520 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.956455 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:42.972278 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.972255 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-31407e\"" Oct 30 06:17:42.972401 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.972348 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd"] Oct 30 06:17:42.988581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:42.988560 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-31407e-dockercfg-dzshj\"" Oct 30 06:17:43.004326 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.004296 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" event={"ID":"c62893d3-85ac-42d9-a5f4-e7eba0aa5924","Type":"ContainerStarted","Data":"9b5f96fb0b44c5469f258edf914b662dec7c36317e3317104dcbe8b66c5520d9"} Oct 30 06:17:43.007266 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.007240 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" event={"ID":"8f277c85-c9a4-47ed-9c18-96fcdf78728e","Type":"ContainerStarted","Data":"c709d5cc48deb96b2f6c2c4055c3dbab5c54ccf1d594be4820f555e2bfdab962"} Oct 30 06:17:43.007384 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.007304 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" Oct 30 06:17:43.064071 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.063980 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" podStartSLOduration=1.914866175 podStartE2EDuration="4.063964928s" totalImagesPullingTime="2.149098753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:43.061706684 +0000 UTC m=+1429.570726571" watchObservedRunningTime="2026-10-30 06:17:43.063964928 +0000 UTC m=+1429.572984815" Oct 30 06:17:43.070477 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.070450 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwz5v\" (UniqueName: \"kubernetes.io/projected/47351595-ca08-49ed-bd29-d8046678774a-kube-api-access-cwz5v\") pod \"payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd\" (UID: \"47351595-ca08-49ed-bd29-d8046678774a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" Oct 30 06:17:43.070679 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.070656 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47351595-ca08-49ed-bd29-d8046678774a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd\" (UID: \"47351595-ca08-49ed-bd29-d8046678774a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" Oct 30 06:17:43.108933 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.108884 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" podStartSLOduration=2.081608091 podStartE2EDuration="4.108871753s" totalImagesPullingTime="2.027263662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:43.108552973 +0000 UTC m=+1429.617572861" watchObservedRunningTime="2026-10-30 06:17:43.108871753 +0000 UTC m=+1429.617891639" Oct 30 06:17:43.172366 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.172333 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47351595-ca08-49ed-bd29-d8046678774a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd\" (UID: \"47351595-ca08-49ed-bd29-d8046678774a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" Oct 30 06:17:43.172580 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.172546 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwz5v\" (UniqueName: \"kubernetes.io/projected/47351595-ca08-49ed-bd29-d8046678774a-kube-api-access-cwz5v\") pod \"payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd\" (UID: \"47351595-ca08-49ed-bd29-d8046678774a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" Oct 30 06:17:43.173985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.173957 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47351595-ca08-49ed-bd29-d8046678774a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd\" (UID: \"47351595-ca08-49ed-bd29-d8046678774a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" Oct 30 06:17:43.213580 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.213554 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwz5v\" (UniqueName: \"kubernetes.io/projected/47351595-ca08-49ed-bd29-d8046678774a-kube-api-access-cwz5v\") pod \"payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd\" (UID: \"47351595-ca08-49ed-bd29-d8046678774a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" Oct 30 06:17:43.274437 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.274405 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" Oct 30 06:17:43.616128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.616038 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa2534f-7589-4a20-b321-10df418b86c5" path="/var/lib/kubelet/pods/0fa2534f-7589-4a20-b321-10df418b86c5/volumes" Oct 30 06:17:43.616820 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.616793 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43830d5c-1f9d-4699-a0f8-089f40e1e098" path="/var/lib/kubelet/pods/43830d5c-1f9d-4699-a0f8-089f40e1e098/volumes" Oct 30 06:17:43.617666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.617645 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6109420f-b8c7-4f71-8796-5e79cdcd53b0" path="/var/lib/kubelet/pods/6109420f-b8c7-4f71-8796-5e79cdcd53b0/volumes" Oct 30 06:17:43.619501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.619482 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd16456-e1e8-4295-8d4a-3045f9e20db1" path="/var/lib/kubelet/pods/bbd16456-e1e8-4295-8d4a-3045f9e20db1/volumes" Oct 30 06:17:43.657615 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:43.657583 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd"] Oct 30 06:17:44.011324 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.011201 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:44.019894 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.019861 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" event={"ID":"47351595-ca08-49ed-bd29-d8046678774a","Type":"ContainerStarted","Data":"87560a0f159be7b7cd585af21bfc8b487e323bd51fa7d7e957fd422d9a88a3c1"} Oct 30 06:17:44.020046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.019918 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" event={"ID":"47351595-ca08-49ed-bd29-d8046678774a","Type":"ContainerStarted","Data":"eede63a0b23cb9b5bb11299844744bdcdf83bf015350fe7df81c92bbdfec497f"} Oct 30 06:17:44.020327 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.020309 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" Oct 30 06:17:44.020612 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.020592 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" Oct 30 06:17:44.063105 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.063053 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" podStartSLOduration=2.063041259 podStartE2EDuration="2.063041259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:44.062441154 +0000 UTC m=+1430.571461044" watchObservedRunningTime="2026-10-30 06:17:44.063041259 +0000 UTC m=+1430.572061145" Oct 30 06:17:44.176881 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.176843 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:44.330496 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.330416 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:44.438804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.438760 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf"] Oct 30 06:17:44.454756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.444006 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" Oct 30 06:17:44.469235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.462161 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf"] Oct 30 06:17:44.469559 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.469540 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-584c9d-dockercfg-4mj86\"" Oct 30 06:17:44.469735 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.469723 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-584c9d\"" Oct 30 06:17:44.605199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.605122 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c24e78-9db8-4c16-b29a-874b93e894dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf\" (UID: \"93c24e78-9db8-4c16-b29a-874b93e894dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" Oct 30 06:17:44.605199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.605175 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr7bj\" (UniqueName: \"kubernetes.io/projected/93c24e78-9db8-4c16-b29a-874b93e894dd-kube-api-access-wr7bj\") pod \"payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf\" (UID: \"93c24e78-9db8-4c16-b29a-874b93e894dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" Oct 30 06:17:44.605840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.605813 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b"] Oct 30 06:17:44.609982 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.609963 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" Oct 30 06:17:44.631434 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.631400 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b"] Oct 30 06:17:44.707316 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.707275 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c24e78-9db8-4c16-b29a-874b93e894dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf\" (UID: \"93c24e78-9db8-4c16-b29a-874b93e894dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" Oct 30 06:17:44.707499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.707337 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr7bj\" (UniqueName: \"kubernetes.io/projected/93c24e78-9db8-4c16-b29a-874b93e894dd-kube-api-access-wr7bj\") pod \"payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf\" (UID: \"93c24e78-9db8-4c16-b29a-874b93e894dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" Oct 30 06:17:44.707499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.707481 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d39900-c6a7-48e9-8437-725d2bc7f918-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b\" (UID: \"75d39900-c6a7-48e9-8437-725d2bc7f918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" Oct 30 06:17:44.707582 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.707527 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qmg5\" (UniqueName: \"kubernetes.io/projected/75d39900-c6a7-48e9-8437-725d2bc7f918-kube-api-access-4qmg5\") pod \"payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b\" (UID: \"75d39900-c6a7-48e9-8437-725d2bc7f918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" Oct 30 06:17:44.708275 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.708250 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c24e78-9db8-4c16-b29a-874b93e894dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf\" (UID: \"93c24e78-9db8-4c16-b29a-874b93e894dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" Oct 30 06:17:44.733318 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.733292 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr7bj\" (UniqueName: \"kubernetes.io/projected/93c24e78-9db8-4c16-b29a-874b93e894dd-kube-api-access-wr7bj\") pod \"payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf\" (UID: \"93c24e78-9db8-4c16-b29a-874b93e894dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" Oct 30 06:17:44.801102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.801067 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" Oct 30 06:17:44.808877 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.808855 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d39900-c6a7-48e9-8437-725d2bc7f918-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b\" (UID: \"75d39900-c6a7-48e9-8437-725d2bc7f918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" Oct 30 06:17:44.808965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.808925 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qmg5\" (UniqueName: \"kubernetes.io/projected/75d39900-c6a7-48e9-8437-725d2bc7f918-kube-api-access-4qmg5\") pod \"payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b\" (UID: \"75d39900-c6a7-48e9-8437-725d2bc7f918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" Oct 30 06:17:44.809583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.809567 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d39900-c6a7-48e9-8437-725d2bc7f918-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b\" (UID: \"75d39900-c6a7-48e9-8437-725d2bc7f918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" Oct 30 06:17:44.833125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.833099 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qmg5\" (UniqueName: \"kubernetes.io/projected/75d39900-c6a7-48e9-8437-725d2bc7f918-kube-api-access-4qmg5\") pod \"payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b\" (UID: \"75d39900-c6a7-48e9-8437-725d2bc7f918\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" Oct 30 06:17:44.929065 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:44.929031 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" Oct 30 06:17:45.114954 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:45.114925 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf"] Oct 30 06:17:45.115446 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:17:45.115421 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93c24e78_9db8_4c16_b29a_874b93e894dd.slice/crio-05300a8989cc7fb7dd9aaa0ddee8dd4fa0c3897a7dd9de95cd1252cc4e11cbfc WatchSource:0}: Error finding container 05300a8989cc7fb7dd9aaa0ddee8dd4fa0c3897a7dd9de95cd1252cc4e11cbfc: Status 404 returned error can't find the container with id 05300a8989cc7fb7dd9aaa0ddee8dd4fa0c3897a7dd9de95cd1252cc4e11cbfc Oct 30 06:17:45.274188 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:45.274155 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b"] Oct 30 06:17:45.279504 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:17:45.278931 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d39900_c6a7_48e9_8437_725d2bc7f918.slice/crio-f38fc2ec439ad0c0fe0030a5acb1a628780227dd0dd75bf5c70bad9691a84680 WatchSource:0}: Error finding container f38fc2ec439ad0c0fe0030a5acb1a628780227dd0dd75bf5c70bad9691a84680: Status 404 returned error can't find the container with id f38fc2ec439ad0c0fe0030a5acb1a628780227dd0dd75bf5c70bad9691a84680 Oct 30 06:17:45.397544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:45.397474 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:46.036734 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:46.036695 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" event={"ID":"75d39900-c6a7-48e9-8437-725d2bc7f918","Type":"ContainerStarted","Data":"c6fabe265533b354fa0b504528549c4cb08ac4c53613816019c3abc818fb7b99"} Oct 30 06:17:46.036952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:46.036798 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" Oct 30 06:17:46.036952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:46.036825 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" event={"ID":"75d39900-c6a7-48e9-8437-725d2bc7f918","Type":"ContainerStarted","Data":"f38fc2ec439ad0c0fe0030a5acb1a628780227dd0dd75bf5c70bad9691a84680"} Oct 30 06:17:46.038910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:46.038881 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" event={"ID":"93c24e78-9db8-4c16-b29a-874b93e894dd","Type":"ContainerStarted","Data":"e2e3b9b522867ab47569c3e852ef55666f610846636fe654f251beb5e41b65ef"} Oct 30 06:17:46.039067 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:46.038933 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" event={"ID":"93c24e78-9db8-4c16-b29a-874b93e894dd","Type":"ContainerStarted","Data":"05300a8989cc7fb7dd9aaa0ddee8dd4fa0c3897a7dd9de95cd1252cc4e11cbfc"} Oct 30 06:17:46.039067 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:46.038960 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" Oct 30 06:17:46.081864 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:46.081803 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" podStartSLOduration=2.081783708 podStartE2EDuration="2.081783708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:46.078059758 +0000 UTC m=+1432.587079674" watchObservedRunningTime="2026-10-30 06:17:46.081783708 +0000 UTC m=+1432.590803601" Oct 30 06:17:46.121687 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:46.121627 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" podStartSLOduration=2.12160969 podStartE2EDuration="2.12160969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:46.118509091 +0000 UTC m=+1432.627528982" watchObservedRunningTime="2026-10-30 06:17:46.12160969 +0000 UTC m=+1432.630629579" Oct 30 06:17:47.167559 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:47.167530 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:49.043538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.043512 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:49.271248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.271178 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:49.441822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.441792 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:49.599827 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.599794 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc"] Oct 30 06:17:49.604096 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.604073 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.627358 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.627334 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0fab6d-openshift-default-dockercfg-7pz45\"" Oct 30 06:17:49.647486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.647457 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc"] Oct 30 06:17:49.664500 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.664456 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:49.670261 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.670197 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.670378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.670273 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.670423 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.670389 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff75821-8a85-4c01-a62f-f000b1c4d98b-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.670486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.670421 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.670486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.670448 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.670626 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.670483 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.670626 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.670585 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.670700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.670625 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrb48\" (UniqueName: \"kubernetes.io/projected/dff75821-8a85-4c01-a62f-f000b1c4d98b-kube-api-access-wrb48\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.670700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.670686 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.771701 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.771626 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff75821-8a85-4c01-a62f-f000b1c4d98b-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.771701 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.771662 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.771701 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.771692 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.771957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.771864 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.772056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.771956 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.772056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.771991 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrb48\" (UniqueName: \"kubernetes.io/projected/dff75821-8a85-4c01-a62f-f000b1c4d98b-kube-api-access-wrb48\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.772056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.772043 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.772419 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.772063 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.772419 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.772276 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.772419 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.772330 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.772419 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.772354 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.772631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.772432 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.772631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.772463 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff75821-8a85-4c01-a62f-f000b1c4d98b-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.772946 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.772925 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.774523 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.774502 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.774628 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.774559 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.795201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.795174 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.795326 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.795289 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrb48\" (UniqueName: \"kubernetes.io/projected/dff75821-8a85-4c01-a62f-f000b1c4d98b-kube-api-access-wrb48\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:49.919168 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:49.919136 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:50.240300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:50.240274 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc"] Oct 30 06:17:50.242049 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:17:50.242022 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff75821_8a85_4c01_a62f_f000b1c4d98b.slice/crio-c6dc2011549098b2582439f38caca744cabe81408dcf0537b3fb4895f4b0b979 WatchSource:0}: Error finding container c6dc2011549098b2582439f38caca744cabe81408dcf0537b3fb4895f4b0b979: Status 404 returned error can't find the container with id c6dc2011549098b2582439f38caca744cabe81408dcf0537b3fb4895f4b0b979 Oct 30 06:17:50.244792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:50.244755 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:50.244871 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:50.244837 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:50.244921 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:50.244881 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:17:51.060792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:51.060753 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:51.079583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:51.079542 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" event={"ID":"dff75821-8a85-4c01-a62f-f000b1c4d98b","Type":"ContainerStarted","Data":"35ff6379fd378a0e52c0e2c4f8920b4e03d4f714858cc68d921d163148a26750"} Oct 30 06:17:51.079776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:51.079644 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" event={"ID":"dff75821-8a85-4c01-a62f-f000b1c4d98b","Type":"ContainerStarted","Data":"c6dc2011549098b2582439f38caca744cabe81408dcf0537b3fb4895f4b0b979"} Oct 30 06:17:51.161243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:51.148924 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" podStartSLOduration=2.148903958 podStartE2EDuration="2.148903958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:51.14069628 +0000 UTC m=+1437.649716167" watchObservedRunningTime="2026-10-30 06:17:51.148903958 +0000 UTC m=+1437.657923846" Oct 30 06:17:51.206481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:51.206452 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:51.659290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:51.657663 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:51.920194 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:51.920103 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:51.921969 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:51.921938 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 06:17:51.922122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:51.921991 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" podUID="dff75821-8a85-4c01-a62f-f000b1c4d98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 06:17:52.919646 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:52.919603 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 06:17:52.920155 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:52.919665 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" podUID="dff75821-8a85-4c01-a62f-f000b1c4d98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 06:17:53.352302 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:53.349734 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:53.920037 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:53.919995 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 06:17:53.920523 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:53.920053 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" podUID="dff75821-8a85-4c01-a62f-f000b1c4d98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 06:17:54.022170 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:54.022141 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" Oct 30 06:17:54.078162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:54.078119 6999 scope.go:122] "RemoveContainer" containerID="2ffa56c964bc7f5b6ad7705708be16e383efaabe7475617f77f743a4fb1732ba" Oct 30 06:17:54.090274 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:54.090249 6999 scope.go:122] "RemoveContainer" containerID="4b9cb6da74effafa84cd4ea321c6dcbfab221d75e634d48c9fd2b94d62be29e6" Oct 30 06:17:54.116957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:54.116935 6999 scope.go:122] "RemoveContainer" containerID="e3226bfcde8afc25ee2157c5bf6cbb6ff002c226741c389e1d266308f8b7d211" Oct 30 06:17:54.125665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:54.125643 6999 scope.go:122] "RemoveContainer" containerID="ae54d1ab9e7efd07da317134084c05b6dd77ed95d56561b05771f0a5d11cf4a5" Oct 30 06:17:54.919643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:54.919553 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 06:17:54.919643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:54.919614 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" podUID="dff75821-8a85-4c01-a62f-f000b1c4d98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 06:17:55.030694 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:55.030667 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" Oct 30 06:17:55.031319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:55.031303 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" Oct 30 06:17:55.272862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:55.272779 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:55.920052 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:55.920012 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 06:17:55.920262 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:55.920068 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" podUID="dff75821-8a85-4c01-a62f-f000b1c4d98b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 06:17:56.924658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:56.924626 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:56.925028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:56.924928 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:56.925608 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:56.925593 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:17:57.052283 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:57.052255 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" Oct 30 06:17:57.054383 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:57.054357 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" Oct 30 06:17:57.189391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:57.189300 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:59.095733 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:59.095696 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:59.411738 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:59.411700 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:59.557110 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:59.557081 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:59.707664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:59.707570 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:17:59.878845 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:17:59.878814 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:00.333771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:00.333736 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:00.545077 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:00.545047 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:00.685698 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:00.685669 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:01.474028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.474000 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz"] Oct 30 06:18:01.480818 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.480796 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:01.496567 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.496546 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-31407e-serving-cert\"" Oct 30 06:18:01.525568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.525532 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz"] Oct 30 06:18:01.581571 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.581536 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7de5845-fa83-4548-9479-dbacf746d212-openshift-service-ca\") pod \"maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:01.581730 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.581647 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7de5845-fa83-4548-9479-dbacf746d212-maas-api-tls\") pod \"maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:01.581798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.581738 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7de5845-fa83-4548-9479-dbacf746d212-metrics-certs\") pod \"maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:01.581857 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.581799 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpxtg\" (UniqueName: \"kubernetes.io/projected/c7de5845-fa83-4548-9479-dbacf746d212-kube-api-access-dpxtg\") pod \"maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:01.674767 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.674736 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:01.683546 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.683505 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7de5845-fa83-4548-9479-dbacf746d212-openshift-service-ca\") pod \"maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:01.683691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.683616 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7de5845-fa83-4548-9479-dbacf746d212-maas-api-tls\") pod \"maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:01.683933 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.683691 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7de5845-fa83-4548-9479-dbacf746d212-metrics-certs\") pod \"maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:01.683933 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.683806 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpxtg\" (UniqueName: \"kubernetes.io/projected/c7de5845-fa83-4548-9479-dbacf746d212-kube-api-access-dpxtg\") pod \"maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:01.684994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.684971 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7de5845-fa83-4548-9479-dbacf746d212-openshift-service-ca\") pod \"maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:01.686618 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.686595 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7de5845-fa83-4548-9479-dbacf746d212-metrics-certs\") pod \"maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:01.686690 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.686672 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7de5845-fa83-4548-9479-dbacf746d212-maas-api-tls\") pod \"maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:01.709927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.709901 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpxtg\" (UniqueName: \"kubernetes.io/projected/c7de5845-fa83-4548-9479-dbacf746d212-kube-api-access-dpxtg\") pod \"maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:01.796373 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.796292 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:01.819874 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:01.819827 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:02.131644 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:02.131601 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz"] Oct 30 06:18:02.133354 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:18:02.133315 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7de5845_fa83_4548_9479_dbacf746d212.slice/crio-e11dbe76ce83a182fbf86fa3a7d574745146cb12f0fe228bdde80be178c86311 WatchSource:0}: Error finding container e11dbe76ce83a182fbf86fa3a7d574745146cb12f0fe228bdde80be178c86311: Status 404 returned error can't find the container with id e11dbe76ce83a182fbf86fa3a7d574745146cb12f0fe228bdde80be178c86311 Oct 30 06:18:02.179716 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:02.179677 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" event={"ID":"c7de5845-fa83-4548-9479-dbacf746d212","Type":"ContainerStarted","Data":"e11dbe76ce83a182fbf86fa3a7d574745146cb12f0fe228bdde80be178c86311"} Oct 30 06:18:02.238608 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:02.238577 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:02.389854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:02.387925 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:03.187462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:03.187428 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" event={"ID":"c7de5845-fa83-4548-9479-dbacf746d212","Type":"ContainerStarted","Data":"a0574456e9dc9057330b9f4032553873252e3d86fe839905352f224056d28b14"} Oct 30 06:18:03.187891 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:03.187532 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:03.235793 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:03.235739 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" podStartSLOduration=1.8039696790000002 podStartE2EDuration="2.235723809s" totalImagesPullingTime="431.75413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:03.235445139 +0000 UTC m=+1449.744465030" watchObservedRunningTime="2026-10-30 06:18:03.235723809 +0000 UTC m=+1449.744743695" Oct 30 06:18:03.655385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:03.655353 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:04.142771 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:18:04.142731 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:18:04.863547 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:04.863517 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:05.053657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:05.053622 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:05.790774 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:05.790744 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:06.492765 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:06.492724 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:06.631142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:06.631115 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:06.799311 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:06.798388 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:06.961170 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:06.961134 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:08.548773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:08.548733 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:08.727401 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:08.727360 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:09.200330 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:09.200303 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:18:10.465691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:10.465657 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:12.266731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:12.266697 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:12.418882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:12.418855 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:14.200830 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:14.200804 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:16.030507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:16.030475 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:18.021405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:18.021358 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:19.337549 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:19.337504 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:19.872362 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:19.872326 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv"] Oct 30 06:18:19.879026 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:19.878999 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" Oct 30 06:18:19.896758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:19.896718 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv"] Oct 30 06:18:19.905792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:19.905761 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f50f6e\"" Oct 30 06:18:19.905970 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:19.905884 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f50f6e-dockercfg-gw6fl\"" Oct 30 06:18:19.978544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:19.978509 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr5bp\" (UniqueName: \"kubernetes.io/projected/069b1c9a-1033-46f7-a37b-74a644d6a069-kube-api-access-pr5bp\") pod \"payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv\" (UID: \"069b1c9a-1033-46f7-a37b-74a644d6a069\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" Oct 30 06:18:19.978734 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:19.978553 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069b1c9a-1033-46f7-a37b-74a644d6a069-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv\" (UID: \"069b1c9a-1033-46f7-a37b-74a644d6a069\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" Oct 30 06:18:20.080138 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:20.080086 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr5bp\" (UniqueName: \"kubernetes.io/projected/069b1c9a-1033-46f7-a37b-74a644d6a069-kube-api-access-pr5bp\") pod \"payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv\" (UID: \"069b1c9a-1033-46f7-a37b-74a644d6a069\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" Oct 30 06:18:20.080372 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:20.080185 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069b1c9a-1033-46f7-a37b-74a644d6a069-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv\" (UID: \"069b1c9a-1033-46f7-a37b-74a644d6a069\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" Oct 30 06:18:20.081124 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:20.081097 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069b1c9a-1033-46f7-a37b-74a644d6a069-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv\" (UID: \"069b1c9a-1033-46f7-a37b-74a644d6a069\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" Oct 30 06:18:20.104204 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:20.104171 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr5bp\" (UniqueName: \"kubernetes.io/projected/069b1c9a-1033-46f7-a37b-74a644d6a069-kube-api-access-pr5bp\") pod \"payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv\" (UID: \"069b1c9a-1033-46f7-a37b-74a644d6a069\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" Oct 30 06:18:20.196534 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:20.196389 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" Oct 30 06:18:20.568026 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:20.567980 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv"] Oct 30 06:18:20.570800 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:18:20.570768 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069b1c9a_1033_46f7_a37b_74a644d6a069.slice/crio-4f5c65cfda449462add6099b91620afa7b47772be6729b897d96359df88647e8 WatchSource:0}: Error finding container 4f5c65cfda449462add6099b91620afa7b47772be6729b897d96359df88647e8: Status 404 returned error can't find the container with id 4f5c65cfda449462add6099b91620afa7b47772be6729b897d96359df88647e8 Oct 30 06:18:20.766580 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:20.766550 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:21.322663 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:21.322622 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" event={"ID":"069b1c9a-1033-46f7-a37b-74a644d6a069","Type":"ContainerStarted","Data":"06f9754fbb6645c9461334551cbdeef750f575312f6b5b5ea0fb9a44f99c0d30"} Oct 30 06:18:21.322989 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:21.322968 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" event={"ID":"069b1c9a-1033-46f7-a37b-74a644d6a069","Type":"ContainerStarted","Data":"4f5c65cfda449462add6099b91620afa7b47772be6729b897d96359df88647e8"} Oct 30 06:18:21.323564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:21.323204 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" Oct 30 06:18:21.368845 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:21.368783 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" podStartSLOduration=2.368762132 podStartE2EDuration="2.368762132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:21.364921971 +0000 UTC m=+1467.873941860" watchObservedRunningTime="2026-10-30 06:18:21.368762132 +0000 UTC m=+1467.877782022" Oct 30 06:18:22.363071 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:22.363036 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:24.178326 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:24.178292 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:26.524997 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:26.524949 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:28.347978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:28.347948 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:28.497919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:28.497883 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:30.306184 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:30.306145 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:32.062073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.062041 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll"] Oct 30 06:18:32.066884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.066864 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:32.099786 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.094820 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8272af-serving-cert\"" Oct 30 06:18:32.099961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.099863 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll"] Oct 30 06:18:32.187174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.187144 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:32.245307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.245278 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928adf71-31d6-41fa-88eb-159cc0545a8c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:32.245486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.245452 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/928adf71-31d6-41fa-88eb-159cc0545a8c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:32.245540 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.245525 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk24r\" (UniqueName: \"kubernetes.io/projected/928adf71-31d6-41fa-88eb-159cc0545a8c-kube-api-access-gk24r\") pod \"maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:32.245687 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.245637 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928adf71-31d6-41fa-88eb-159cc0545a8c-metrics-certs\") pod \"maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:32.333967 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.333869 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" Oct 30 06:18:32.338254 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.338185 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:32.346962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.346929 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk24r\" (UniqueName: \"kubernetes.io/projected/928adf71-31d6-41fa-88eb-159cc0545a8c-kube-api-access-gk24r\") pod \"maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:32.347144 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.347068 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928adf71-31d6-41fa-88eb-159cc0545a8c-metrics-certs\") pod \"maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:32.347144 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.347108 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928adf71-31d6-41fa-88eb-159cc0545a8c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:32.347329 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.347273 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/928adf71-31d6-41fa-88eb-159cc0545a8c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:32.347907 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.347877 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/928adf71-31d6-41fa-88eb-159cc0545a8c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:32.350126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.350100 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928adf71-31d6-41fa-88eb-159cc0545a8c-metrics-certs\") pod \"maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:32.350276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.350101 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928adf71-31d6-41fa-88eb-159cc0545a8c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:32.370493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.370463 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk24r\" (UniqueName: \"kubernetes.io/projected/928adf71-31d6-41fa-88eb-159cc0545a8c-kube-api-access-gk24r\") pod \"maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:32.380700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.380677 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:32.508393 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.508362 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:32.708953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:32.708929 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll"] Oct 30 06:18:32.710180 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:18:32.710151 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928adf71_31d6_41fa_88eb_159cc0545a8c.slice/crio-4e857fbb83ed9a6803863736086fdb422391eeabbe3fda42feb998d67f856f27 WatchSource:0}: Error finding container 4e857fbb83ed9a6803863736086fdb422391eeabbe3fda42feb998d67f856f27: Status 404 returned error can't find the container with id 4e857fbb83ed9a6803863736086fdb422391eeabbe3fda42feb998d67f856f27 Oct 30 06:18:33.407466 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:33.407427 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" event={"ID":"928adf71-31d6-41fa-88eb-159cc0545a8c","Type":"ContainerStarted","Data":"5bfd740d7fa3be4e068de75c63ae5efd60e37fe37beacde5c4fc3a2e38c8e750"} Oct 30 06:18:33.407950 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:33.407530 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" event={"ID":"928adf71-31d6-41fa-88eb-159cc0545a8c","Type":"ContainerStarted","Data":"4e857fbb83ed9a6803863736086fdb422391eeabbe3fda42feb998d67f856f27"} Oct 30 06:18:33.408011 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:33.407987 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:33.453418 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:33.453359 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" podStartSLOduration=1.119319474 podStartE2EDuration="1.453340217s" totalImagesPullingTime="334.020743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:33.451862085 +0000 UTC m=+1479.960881972" watchObservedRunningTime="2026-10-30 06:18:33.453340217 +0000 UTC m=+1479.962360106" Oct 30 06:18:34.142246 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:18:34.142190 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:18:34.176913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.176883 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:34.734754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.734722 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf"] Oct 30 06:18:34.740122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.740095 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.769092 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.768889 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf"] Oct 30 06:18:34.881316 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.881284 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.881316 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.881319 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.881517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.881344 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.881517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.881364 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.881517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.881480 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.881645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.881530 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.881719 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.881701 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.881767 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.881748 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.881819 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.881776 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfghp\" (UniqueName: \"kubernetes.io/projected/255d0ee6-d1a8-4ccc-a9b9-79343540f145-kube-api-access-lfghp\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.984114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.984073 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.984316 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.984142 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.984406 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.984330 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.984406 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.984384 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.984524 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.984423 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfghp\" (UniqueName: \"kubernetes.io/projected/255d0ee6-d1a8-4ccc-a9b9-79343540f145-kube-api-access-lfghp\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.984524 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.984480 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.984524 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.984506 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.984682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.984548 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.984682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.984576 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.985103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.985018 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.985347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.985317 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.985490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.985373 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.985648 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.985626 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.985913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.985888 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.987777 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.987745 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:34.987921 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:34.987905 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:35.013447 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:35.013418 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfghp\" (UniqueName: \"kubernetes.io/projected/255d0ee6-d1a8-4ccc-a9b9-79343540f145-kube-api-access-lfghp\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:35.013556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:35.013425 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:35.061034 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:35.061011 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:35.381836 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:35.381813 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf"] Oct 30 06:18:35.383300 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:18:35.383274 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod255d0ee6_d1a8_4ccc_a9b9_79343540f145.slice/crio-dfb16eac7a0dd89eb46fec29b5181daed60df5a55a1ec81aa9bb624713235204 WatchSource:0}: Error finding container dfb16eac7a0dd89eb46fec29b5181daed60df5a55a1ec81aa9bb624713235204: Status 404 returned error can't find the container with id dfb16eac7a0dd89eb46fec29b5181daed60df5a55a1ec81aa9bb624713235204 Oct 30 06:18:35.386237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:35.386179 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:35.386307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:35.386289 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:35.386356 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:35.386341 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:35.425293 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:35.425267 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" event={"ID":"255d0ee6-d1a8-4ccc-a9b9-79343540f145","Type":"ContainerStarted","Data":"dfb16eac7a0dd89eb46fec29b5181daed60df5a55a1ec81aa9bb624713235204"} Oct 30 06:18:36.446240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:36.443433 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" event={"ID":"255d0ee6-d1a8-4ccc-a9b9-79343540f145","Type":"ContainerStarted","Data":"281ff6c9e1b6571c78ce9a92671579c49cc80fab5772d2074ef02c3caf389bd2"} Oct 30 06:18:36.525354 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:36.525295 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" podStartSLOduration=2.525275587 podStartE2EDuration="2.525275587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:36.507775279 +0000 UTC m=+1483.016795166" watchObservedRunningTime="2026-10-30 06:18:36.525275587 +0000 UTC m=+1483.034295476" Oct 30 06:18:36.893940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:36.893912 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:37.037748 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:37.037708 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:37.061676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:37.061639 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:37.064291 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:37.063913 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 06:18:37.064291 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:37.063966 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" podUID="255d0ee6-d1a8-4ccc-a9b9-79343540f145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 06:18:37.385137 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:37.385098 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:37.612756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:37.612732 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:38.063821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:38.063788 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 06:18:38.064106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:38.064076 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" podUID="255d0ee6-d1a8-4ccc-a9b9-79343540f145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 06:18:38.971404 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:38.971366 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:39.075242 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:39.070619 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 06:18:39.075242 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:39.070684 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" podUID="255d0ee6-d1a8-4ccc-a9b9-79343540f145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 06:18:39.420054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:39.420022 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:18:40.061702 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:40.061667 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 06:18:40.062187 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:40.061720 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" podUID="255d0ee6-d1a8-4ccc-a9b9-79343540f145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 06:18:40.961398 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:40.961361 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:41.065650 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:41.065615 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 06:18:41.066309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:41.066268 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" podUID="255d0ee6-d1a8-4ccc-a9b9-79343540f145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 06:18:42.065824 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:42.065793 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:42.066304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:42.066138 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:42.066947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:42.066930 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:18:42.965787 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:42.965746 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:43.134607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:43.134568 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:44.726181 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:44.726154 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:44.876111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:44.876062 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:45.085667 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:45.085555 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:45.307920 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:45.307890 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:45.534255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:45.534208 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:45.981486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:45.981446 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:46.145945 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:46.145896 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:46.298304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:46.298181 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:47.110553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:47.110519 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:49.516174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.516144 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:49.759840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.759806 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl"] Oct 30 06:18:49.763360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.763336 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.790266 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.790233 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl"] Oct 30 06:18:49.846174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.846139 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0576b919-84c1-494c-b84a-021af83ccde7-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.846391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.846180 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tpp5\" (UniqueName: \"kubernetes.io/projected/0576b919-84c1-494c-b84a-021af83ccde7-kube-api-access-2tpp5\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.846391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.846204 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.846525 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.846451 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.846525 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.846503 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.846604 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.846554 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0576b919-84c1-494c-b84a-021af83ccde7-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.846673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.846657 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.846722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.846680 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0576b919-84c1-494c-b84a-021af83ccde7-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.846787 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.846776 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.947693 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.947657 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.947876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.947755 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0576b919-84c1-494c-b84a-021af83ccde7-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.947876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.947780 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tpp5\" (UniqueName: \"kubernetes.io/projected/0576b919-84c1-494c-b84a-021af83ccde7-kube-api-access-2tpp5\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.947876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.947802 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.948045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.947888 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.948045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.947921 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.948045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.947946 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0576b919-84c1-494c-b84a-021af83ccde7-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.948045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.948004 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.948045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.948027 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0576b919-84c1-494c-b84a-021af83ccde7-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.948371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.948284 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-workload-certs\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.948513 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.948458 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-workload-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.948656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.948520 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-credential-socket\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.948656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.948493 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-istio-data\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.948656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.948611 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0576b919-84c1-494c-b84a-021af83ccde7-istiod-ca-cert\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.950319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.950298 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-istio-envoy\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.950619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.950601 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0576b919-84c1-494c-b84a-021af83ccde7-istio-podinfo\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.971458 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.971427 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0576b919-84c1-494c-b84a-021af83ccde7-istio-token\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:49.971611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:49.971496 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tpp5\" (UniqueName: \"kubernetes.io/projected/0576b919-84c1-494c-b84a-021af83ccde7-kube-api-access-2tpp5\") pod \"e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:50.077622 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:50.077528 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:50.404947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:50.404906 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl"] Oct 30 06:18:50.409069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:50.409012 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:50.409360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:50.409335 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:50.409550 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:50.409527 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:18:50.610023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:50.609980 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" event={"ID":"0576b919-84c1-494c-b84a-021af83ccde7","Type":"ContainerStarted","Data":"ae86c6aec0d707c55e66a02112b3a943ae165e6e0c6e1527c565ad2ddc287686"} Oct 30 06:18:50.610541 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:50.610043 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" event={"ID":"0576b919-84c1-494c-b84a-021af83ccde7","Type":"ContainerStarted","Data":"91271e0be3949ef6c1777e8def9f8cd5256e11b180238ad8035d56661b097f39"} Oct 30 06:18:50.656745 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:50.656647 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" podStartSLOduration=1.6566349 podStartE2EDuration="1.6566349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:50.654460726 +0000 UTC m=+1497.163480614" watchObservedRunningTime="2026-10-30 06:18:50.6566349 +0000 UTC m=+1497.165654783" Oct 30 06:18:51.078781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:51.078698 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:51.080722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:51.080697 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 06:18:51.080872 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:51.080747 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" podUID="0576b919-84c1-494c-b84a-021af83ccde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 06:18:51.275157 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:51.275061 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:52.078915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:52.078882 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 06:18:52.079384 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:52.078938 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" podUID="0576b919-84c1-494c-b84a-021af83ccde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 06:18:53.078776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:53.078739 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 06:18:53.078961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:53.078795 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" podUID="0576b919-84c1-494c-b84a-021af83ccde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 06:18:53.093193 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:53.093158 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:54.078669 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:54.078635 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 06:18:54.078852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:54.078691 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" podUID="0576b919-84c1-494c-b84a-021af83ccde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 06:18:54.265032 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:54.264999 6999 scope.go:122] "RemoveContainer" containerID="81a2428d1251b7f6c87b393f447b1e67e780b74d697720011e7933e9c9ae83c9" Oct 30 06:18:55.079295 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:55.078556 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 06:18:55.079295 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:55.078616 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" podUID="0576b919-84c1-494c-b84a-021af83ccde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 06:18:55.606602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:55.606565 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:56.078665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:56.078631 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 06:18:56.078847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:56.078680 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" podUID="0576b919-84c1-494c-b84a-021af83ccde7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 06:18:57.082862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:57.082834 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:57.083330 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:57.083314 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:57.084195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:57.084178 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:18:57.451290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:57.451241 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:59.279558 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:59.279530 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:18:59.960632 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:18:59.960601 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:00.662828 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:00.662797 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:01.075764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:01.075688 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:01.231867 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:01.231817 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:01.378735 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:01.378708 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:02.043426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:02.043362 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:03.077253 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:03.075017 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:04.142311 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:19:04.142266 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:19:05.539724 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:05.539694 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:07.336558 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:07.336527 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:08.411484 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:08.411451 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:08.565730 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:08.565700 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:08.708744 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:08.708654 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:09.214015 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:09.213983 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:10.508882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:10.508846 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:10.684566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:10.684534 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:11.574189 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.574150 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn"] Oct 30 06:19:11.579693 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.579669 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:11.613809 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.611006 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-bc5d8f-serving-cert\"" Oct 30 06:19:11.615134 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.615109 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn"] Oct 30 06:19:11.686997 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.686961 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7422fc62-629d-4465-8793-a0821ae7d918-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:11.686997 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.686998 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7422fc62-629d-4465-8793-a0821ae7d918-metrics-certs\") pod \"maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:11.687284 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.687263 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7422fc62-629d-4465-8793-a0821ae7d918-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:11.687434 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.687404 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkhpc\" (UniqueName: \"kubernetes.io/projected/7422fc62-629d-4465-8793-a0821ae7d918-kube-api-access-vkhpc\") pod \"maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:11.701579 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.701553 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:11.788286 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.788256 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7422fc62-629d-4465-8793-a0821ae7d918-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:11.788469 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.788320 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkhpc\" (UniqueName: \"kubernetes.io/projected/7422fc62-629d-4465-8793-a0821ae7d918-kube-api-access-vkhpc\") pod \"maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:11.788469 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.788373 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7422fc62-629d-4465-8793-a0821ae7d918-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:11.788469 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.788389 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7422fc62-629d-4465-8793-a0821ae7d918-metrics-certs\") pod \"maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:11.789029 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.789004 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7422fc62-629d-4465-8793-a0821ae7d918-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:11.790908 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.790888 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7422fc62-629d-4465-8793-a0821ae7d918-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:11.791007 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.790956 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7422fc62-629d-4465-8793-a0821ae7d918-metrics-certs\") pod \"maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:11.813947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.813880 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkhpc\" (UniqueName: \"kubernetes.io/projected/7422fc62-629d-4465-8793-a0821ae7d918-kube-api-access-vkhpc\") pod \"maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:11.849683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.849592 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:11.894186 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:11.894154 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:12.210456 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:12.210432 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn"] Oct 30 06:19:12.211712 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:19:12.211684 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7422fc62_629d_4465_8793_a0821ae7d918.slice/crio-1d0f51a97a61b9926a07dc36a3871068c2b4fd175a6bb6ab578fbf9d890fdbfa WatchSource:0}: Error finding container 1d0f51a97a61b9926a07dc36a3871068c2b4fd175a6bb6ab578fbf9d890fdbfa: Status 404 returned error can't find the container with id 1d0f51a97a61b9926a07dc36a3871068c2b4fd175a6bb6ab578fbf9d890fdbfa Oct 30 06:19:12.791400 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:12.791367 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" event={"ID":"7422fc62-629d-4465-8793-a0821ae7d918","Type":"ContainerStarted","Data":"8df5d55ecd6f4d0b99224263fc70b26d815f61be38d3cfab65a1989db36f2b78"} Oct 30 06:19:12.791853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:12.791414 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" event={"ID":"7422fc62-629d-4465-8793-a0821ae7d918","Type":"ContainerStarted","Data":"1d0f51a97a61b9926a07dc36a3871068c2b4fd175a6bb6ab578fbf9d890fdbfa"} Oct 30 06:19:12.791853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:12.791446 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:12.841179 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:12.836451 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" podStartSLOduration=1.394606976 podStartE2EDuration="1.836432221s" totalImagesPullingTime="441.825245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:12.833428753 +0000 UTC m=+1519.342448641" watchObservedRunningTime="2026-10-30 06:19:12.836432221 +0000 UTC m=+1519.345452108" Oct 30 06:19:13.640892 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:13.640866 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:15.515607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:15.515564 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:17.791802 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:17.791770 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:18.805191 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:18.805166 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:19:19.531042 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:19.531008 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:21.380643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:21.380605 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:23.632607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:23.632577 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:25.492979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:25.492946 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:26.806734 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:26.806702 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:26.979627 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:26.979598 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:27.943605 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:27.943569 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:28.349275 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:28.349252 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:28.814498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:28.814461 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:28.986536 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:28.986503 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:29.973116 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:29.973082 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:31.351096 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:31.351063 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:31.531069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:31.531032 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:32.671949 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:32.671917 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:32.899676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:32.899649 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:33.073658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:33.073568 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:34.141850 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:19:34.141768 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:19:34.682654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:34.682602 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:34.845917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:34.845885 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:35.158811 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:35.158783 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:37.020642 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:37.020613 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:38.754703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:38.754671 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:41.046653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:41.046623 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:42.915372 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:42.915331 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:44.464626 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:44.464590 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:44.616281 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:44.616249 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:44.768333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:44.768254 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:45.002949 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:45.002916 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:47.549920 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:47.549884 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:49.033336 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.033307 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk"] Oct 30 06:19:49.033773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.033517 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" podUID="179d1033-3d70-4cca-a7cc-0a6bc3223623" containerName="maas-api" containerID="cri-o://1bb9df3e7157e437305d1062e1552b0db68c8618c1f5ccf682222d51fd2f3dbf" gracePeriod=30 Oct 30 06:19:49.129427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.129399 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:49.443110 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.443037 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:19:49.544239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.543722 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-maas-api-tls\") pod \"179d1033-3d70-4cca-a7cc-0a6bc3223623\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " Oct 30 06:19:49.544239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.543777 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/179d1033-3d70-4cca-a7cc-0a6bc3223623-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/179d1033-3d70-4cca-a7cc-0a6bc3223623-openshift-service-ca\") pod \"179d1033-3d70-4cca-a7cc-0a6bc3223623\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " Oct 30 06:19:49.544239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.543817 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-metrics-certs\" (UniqueName: \"kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-metrics-certs\") pod \"179d1033-3d70-4cca-a7cc-0a6bc3223623\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " Oct 30 06:19:49.544239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.544085 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/179d1033-3d70-4cca-a7cc-0a6bc3223623-kube-api-access-z9wt8\" (UniqueName: \"kubernetes.io/projected/179d1033-3d70-4cca-a7cc-0a6bc3223623-kube-api-access-z9wt8\") pod \"179d1033-3d70-4cca-a7cc-0a6bc3223623\" (UID: \"179d1033-3d70-4cca-a7cc-0a6bc3223623\") " Oct 30 06:19:49.560459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.560412 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-metrics-certs" pod "179d1033-3d70-4cca-a7cc-0a6bc3223623" (UID: "179d1033-3d70-4cca-a7cc-0a6bc3223623"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:19:49.563916 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.563872 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-maas-api-tls" pod "179d1033-3d70-4cca-a7cc-0a6bc3223623" (UID: "179d1033-3d70-4cca-a7cc-0a6bc3223623"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:19:49.568448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.568416 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179d1033-3d70-4cca-a7cc-0a6bc3223623-openshift-service-ca" pod "179d1033-3d70-4cca-a7cc-0a6bc3223623" (UID: "179d1033-3d70-4cca-a7cc-0a6bc3223623"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:49.568967 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.568930 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/179d1033-3d70-4cca-a7cc-0a6bc3223623-kube-api-access-z9wt8" pod "179d1033-3d70-4cca-a7cc-0a6bc3223623" (UID: "179d1033-3d70-4cca-a7cc-0a6bc3223623"). InnerVolumeSpecName "kube-api-access-z9wt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:49.647270 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.647202 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9wt8\" (UniqueName: \"kubernetes.io/projected/179d1033-3d70-4cca-a7cc-0a6bc3223623-kube-api-access-z9wt8\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:19:49.647270 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.647272 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:19:49.647486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.647289 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/179d1033-3d70-4cca-a7cc-0a6bc3223623-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:19:49.647486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.647304 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/179d1033-3d70-4cca-a7cc-0a6bc3223623-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:19:49.745065 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.744969 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:49.884785 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:49.884753 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:50.037878 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:50.037802 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:50.161835 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:50.161803 6999 generic.go:356] "Generic (PLEG): container finished" podID="179d1033-3d70-4cca-a7cc-0a6bc3223623" containerID="1bb9df3e7157e437305d1062e1552b0db68c8618c1f5ccf682222d51fd2f3dbf" exitCode=0 Oct 30 06:19:50.162019 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:50.161849 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" Oct 30 06:19:50.162019 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:50.161894 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" event={"ID":"179d1033-3d70-4cca-a7cc-0a6bc3223623","Type":"ContainerDied","Data":"1bb9df3e7157e437305d1062e1552b0db68c8618c1f5ccf682222d51fd2f3dbf"} Oct 30 06:19:50.162019 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:50.161952 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk" event={"ID":"179d1033-3d70-4cca-a7cc-0a6bc3223623","Type":"ContainerDied","Data":"19151ad007112f856acd7f82c012531a09b9cf32e5d60a89e56fa6b3cb1e9744"} Oct 30 06:19:50.162019 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:50.161965 6999 scope.go:122] "RemoveContainer" containerID="1bb9df3e7157e437305d1062e1552b0db68c8618c1f5ccf682222d51fd2f3dbf" Oct 30 06:19:50.174835 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:50.174812 6999 scope.go:122] "RemoveContainer" containerID="1bb9df3e7157e437305d1062e1552b0db68c8618c1f5ccf682222d51fd2f3dbf" Oct 30 06:19:50.175146 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:19:50.175129 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb9df3e7157e437305d1062e1552b0db68c8618c1f5ccf682222d51fd2f3dbf\": container with ID starting with 1bb9df3e7157e437305d1062e1552b0db68c8618c1f5ccf682222d51fd2f3dbf not found: ID does not exist" containerID="1bb9df3e7157e437305d1062e1552b0db68c8618c1f5ccf682222d51fd2f3dbf" Oct 30 06:19:50.175267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:50.175153 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb9df3e7157e437305d1062e1552b0db68c8618c1f5ccf682222d51fd2f3dbf"} err="failed to get container status \"1bb9df3e7157e437305d1062e1552b0db68c8618c1f5ccf682222d51fd2f3dbf\": rpc error: code = NotFound desc = could not find container \"1bb9df3e7157e437305d1062e1552b0db68c8618c1f5ccf682222d51fd2f3dbf\": container with ID starting with 1bb9df3e7157e437305d1062e1552b0db68c8618c1f5ccf682222d51fd2f3dbf not found: ID does not exist" Oct 30 06:19:50.215248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:50.215206 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk"] Oct 30 06:19:50.225205 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:50.225183 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0fab6d-dbff74659-m6mjk"] Oct 30 06:19:50.417459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:50.417427 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:51.608732 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:51.608702 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="179d1033-3d70-4cca-a7cc-0a6bc3223623" path="/var/lib/kubelet/pods/179d1033-3d70-4cca-a7cc-0a6bc3223623/volumes" Oct 30 06:19:51.856197 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:51.856169 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:54.236391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:54.236359 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:54.299734 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:54.299698 6999 scope.go:122] "RemoveContainer" containerID="c9cd77fd96f1dc7794f933349cec8d044751e1070a2f449526e678f9dd236fc9" Oct 30 06:19:55.444467 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:55.444434 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:55.594693 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:55.594662 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:56.133645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:56.133585 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:56.511777 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:56.511700 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:56.711277 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:56.711243 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:56.830237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:56.830137 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc"] Oct 30 06:19:56.830463 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:56.830430 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" podUID="dff75821-8a85-4c01-a62f-f000b1c4d98b" containerName="istio-proxy" containerID="cri-o://35ff6379fd378a0e52c0e2c4f8920b4e03d4f714858cc68d921d163148a26750" gracePeriod=30 Oct 30 06:19:56.839653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:56.839628 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl"] Oct 30 06:19:56.839871 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:56.839832 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" podUID="0576b919-84c1-494c-b84a-021af83ccde7" containerName="istio-proxy" containerID="cri-o://ae86c6aec0d707c55e66a02112b3a943ae165e6e0c6e1527c565ad2ddc287686" gracePeriod=30 Oct 30 06:19:56.848194 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:56.848169 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf"] Oct 30 06:19:56.848645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:56.848607 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" podUID="255d0ee6-d1a8-4ccc-a9b9-79343540f145" containerName="istio-proxy" containerID="cri-o://281ff6c9e1b6571c78ce9a92671579c49cc80fab5772d2074ef02c3caf389bd2" gracePeriod=30 Oct 30 06:19:56.895751 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:56.895719 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:19:56.937064 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:56.928332 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:19:56.937064 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:56.928456 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" podUID="dff75821-8a85-4c01-a62f-f000b1c4d98b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:19:57.068981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:57.068942 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:19:57.069175 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:57.069008 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" podUID="255d0ee6-d1a8-4ccc-a9b9-79343540f145" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:19:57.084821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:57.084747 6999 patch_prober.go:29] interesting pod/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:19:57.084821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:57.084800 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" podUID="0576b919-84c1-494c-b84a-021af83ccde7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:19:58.629350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:19:58.629310 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:00.093113 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:00.093081 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:00.471287 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:00.471251 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:00.780103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:00.780012 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:02.146338 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.146310 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:20:02.176856 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.176821 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:02.191901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.191883 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:20:02.229812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.229770 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-workload-certs\") pod \"0576b919-84c1-494c-b84a-021af83ccde7\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " Oct 30 06:20:02.230048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.229825 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-workload-socket\") pod \"0576b919-84c1-494c-b84a-021af83ccde7\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " Oct 30 06:20:02.230048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.230005 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0576b919-84c1-494c-b84a-021af83ccde7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0576b919-84c1-494c-b84a-021af83ccde7-istio-podinfo\") pod \"0576b919-84c1-494c-b84a-021af83ccde7\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " Oct 30 06:20:02.230139 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.230047 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-workload-certs" pod "0576b919-84c1-494c-b84a-021af83ccde7" (UID: "0576b919-84c1-494c-b84a-021af83ccde7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.230139 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.230052 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-istio-data\") pod \"0576b919-84c1-494c-b84a-021af83ccde7\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " Oct 30 06:20:02.230243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.230164 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0576b919-84c1-494c-b84a-021af83ccde7-kube-api-access-2tpp5\" (UniqueName: \"kubernetes.io/projected/0576b919-84c1-494c-b84a-021af83ccde7-kube-api-access-2tpp5\") pod \"0576b919-84c1-494c-b84a-021af83ccde7\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " Oct 30 06:20:02.230243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.230171 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-workload-socket" pod "0576b919-84c1-494c-b84a-021af83ccde7" (UID: "0576b919-84c1-494c-b84a-021af83ccde7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.230371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.230322 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0576b919-84c1-494c-b84a-021af83ccde7-istio-token\" (UniqueName: \"kubernetes.io/projected/0576b919-84c1-494c-b84a-021af83ccde7-istio-token\") pod \"0576b919-84c1-494c-b84a-021af83ccde7\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " Oct 30 06:20:02.230430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.230377 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-credential-socket\") pod \"0576b919-84c1-494c-b84a-021af83ccde7\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " Oct 30 06:20:02.230430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.230409 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-istio-envoy\") pod \"0576b919-84c1-494c-b84a-021af83ccde7\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " Oct 30 06:20:02.230543 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.230440 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-istio-data" pod "0576b919-84c1-494c-b84a-021af83ccde7" (UID: "0576b919-84c1-494c-b84a-021af83ccde7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.230543 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.230464 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0576b919-84c1-494c-b84a-021af83ccde7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0576b919-84c1-494c-b84a-021af83ccde7-istiod-ca-cert\") pod \"0576b919-84c1-494c-b84a-021af83ccde7\" (UID: \"0576b919-84c1-494c-b84a-021af83ccde7\") " Oct 30 06:20:02.230663 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.230643 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-credential-socket" pod "0576b919-84c1-494c-b84a-021af83ccde7" (UID: "0576b919-84c1-494c-b84a-021af83ccde7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.231202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.231140 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.231202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.231164 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.231202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.231177 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.231202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.231190 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.231668 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.231516 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0576b919-84c1-494c-b84a-021af83ccde7-istiod-ca-cert" pod "0576b919-84c1-494c-b84a-021af83ccde7" (UID: "0576b919-84c1-494c-b84a-021af83ccde7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:02.233804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.233648 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0576b919-84c1-494c-b84a-021af83ccde7-istio-podinfo" pod "0576b919-84c1-494c-b84a-021af83ccde7" (UID: "0576b919-84c1-494c-b84a-021af83ccde7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:02.233804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.233770 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0576b919-84c1-494c-b84a-021af83ccde7-istio-token" pod "0576b919-84c1-494c-b84a-021af83ccde7" (UID: "0576b919-84c1-494c-b84a-021af83ccde7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.234245 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.234184 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0576b919-84c1-494c-b84a-021af83ccde7-kube-api-access-2tpp5" pod "0576b919-84c1-494c-b84a-021af83ccde7" (UID: "0576b919-84c1-494c-b84a-021af83ccde7"). InnerVolumeSpecName "kube-api-access-2tpp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.234561 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.234527 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-istio-envoy" pod "0576b919-84c1-494c-b84a-021af83ccde7" (UID: "0576b919-84c1-494c-b84a-021af83ccde7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.242728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.242698 6999 generic.go:356] "Generic (PLEG): container finished" podID="dff75821-8a85-4c01-a62f-f000b1c4d98b" containerID="35ff6379fd378a0e52c0e2c4f8920b4e03d4f714858cc68d921d163148a26750" exitCode=0 Oct 30 06:20:02.242824 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.242772 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" event={"ID":"dff75821-8a85-4c01-a62f-f000b1c4d98b","Type":"ContainerDied","Data":"35ff6379fd378a0e52c0e2c4f8920b4e03d4f714858cc68d921d163148a26750"} Oct 30 06:20:02.244506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.244377 6999 generic.go:356] "Generic (PLEG): container finished" podID="255d0ee6-d1a8-4ccc-a9b9-79343540f145" containerID="281ff6c9e1b6571c78ce9a92671579c49cc80fab5772d2074ef02c3caf389bd2" exitCode=0 Oct 30 06:20:02.244506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.244411 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" event={"ID":"255d0ee6-d1a8-4ccc-a9b9-79343540f145","Type":"ContainerDied","Data":"281ff6c9e1b6571c78ce9a92671579c49cc80fab5772d2074ef02c3caf389bd2"} Oct 30 06:20:02.244506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.244437 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" Oct 30 06:20:02.244506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.244458 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf" event={"ID":"255d0ee6-d1a8-4ccc-a9b9-79343540f145","Type":"ContainerDied","Data":"dfb16eac7a0dd89eb46fec29b5181daed60df5a55a1ec81aa9bb624713235204"} Oct 30 06:20:02.244506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.244490 6999 scope.go:122] "RemoveContainer" containerID="281ff6c9e1b6571c78ce9a92671579c49cc80fab5772d2074ef02c3caf389bd2" Oct 30 06:20:02.252728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.252690 6999 generic.go:356] "Generic (PLEG): container finished" podID="0576b919-84c1-494c-b84a-021af83ccde7" containerID="ae86c6aec0d707c55e66a02112b3a943ae165e6e0c6e1527c565ad2ddc287686" exitCode=0 Oct 30 06:20:02.253498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.252809 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" event={"ID":"0576b919-84c1-494c-b84a-021af83ccde7","Type":"ContainerDied","Data":"ae86c6aec0d707c55e66a02112b3a943ae165e6e0c6e1527c565ad2ddc287686"} Oct 30 06:20:02.253498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.252855 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" event={"ID":"0576b919-84c1-494c-b84a-021af83ccde7","Type":"ContainerDied","Data":"91271e0be3949ef6c1777e8def9f8cd5256e11b180238ad8035d56661b097f39"} Oct 30 06:20:02.253498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.252926 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl" Oct 30 06:20:02.276854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.276829 6999 scope.go:122] "RemoveContainer" containerID="281ff6c9e1b6571c78ce9a92671579c49cc80fab5772d2074ef02c3caf389bd2" Oct 30 06:20:02.277301 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:20:02.277273 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281ff6c9e1b6571c78ce9a92671579c49cc80fab5772d2074ef02c3caf389bd2\": container with ID starting with 281ff6c9e1b6571c78ce9a92671579c49cc80fab5772d2074ef02c3caf389bd2 not found: ID does not exist" containerID="281ff6c9e1b6571c78ce9a92671579c49cc80fab5772d2074ef02c3caf389bd2" Oct 30 06:20:02.277402 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.277313 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281ff6c9e1b6571c78ce9a92671579c49cc80fab5772d2074ef02c3caf389bd2"} err="failed to get container status \"281ff6c9e1b6571c78ce9a92671579c49cc80fab5772d2074ef02c3caf389bd2\": rpc error: code = NotFound desc = could not find container \"281ff6c9e1b6571c78ce9a92671579c49cc80fab5772d2074ef02c3caf389bd2\": container with ID starting with 281ff6c9e1b6571c78ce9a92671579c49cc80fab5772d2074ef02c3caf389bd2 not found: ID does not exist" Oct 30 06:20:02.277402 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.277338 6999 scope.go:122] "RemoveContainer" containerID="ae86c6aec0d707c55e66a02112b3a943ae165e6e0c6e1527c565ad2ddc287686" Oct 30 06:20:02.287351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.287291 6999 scope.go:122] "RemoveContainer" containerID="ae86c6aec0d707c55e66a02112b3a943ae165e6e0c6e1527c565ad2ddc287686" Oct 30 06:20:02.287585 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:20:02.287567 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae86c6aec0d707c55e66a02112b3a943ae165e6e0c6e1527c565ad2ddc287686\": container with ID starting with ae86c6aec0d707c55e66a02112b3a943ae165e6e0c6e1527c565ad2ddc287686 not found: ID does not exist" containerID="ae86c6aec0d707c55e66a02112b3a943ae165e6e0c6e1527c565ad2ddc287686" Oct 30 06:20:02.287644 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.287595 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae86c6aec0d707c55e66a02112b3a943ae165e6e0c6e1527c565ad2ddc287686"} err="failed to get container status \"ae86c6aec0d707c55e66a02112b3a943ae165e6e0c6e1527c565ad2ddc287686\": rpc error: code = NotFound desc = could not find container \"ae86c6aec0d707c55e66a02112b3a943ae165e6e0c6e1527c565ad2ddc287686\": container with ID starting with ae86c6aec0d707c55e66a02112b3a943ae165e6e0c6e1527c565ad2ddc287686 not found: ID does not exist" Oct 30 06:20:02.320670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.320608 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl"] Oct 30 06:20:02.323993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.323972 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-v4jkl"] Oct 30 06:20:02.332654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.332628 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istiod-ca-cert\") pod \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " Oct 30 06:20:02.332795 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.332681 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-envoy\") pod \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " Oct 30 06:20:02.332795 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.332735 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-workload-certs\") pod \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " Oct 30 06:20:02.332912 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.332841 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/255d0ee6-d1a8-4ccc-a9b9-79343540f145-kube-api-access-lfghp\" (UniqueName: \"kubernetes.io/projected/255d0ee6-d1a8-4ccc-a9b9-79343540f145-kube-api-access-lfghp\") pod \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " Oct 30 06:20:02.332912 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.332906 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-data\") pod \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " Oct 30 06:20:02.333022 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.332937 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-podinfo\") pod \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " Oct 30 06:20:02.333022 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.332969 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-credential-socket\") pod \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " Oct 30 06:20:02.333022 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.333018 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-workload-socket\") pod \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " Oct 30 06:20:02.333171 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.333159 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-token\" (UniqueName: \"kubernetes.io/projected/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-token\") pod \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\" (UID: \"255d0ee6-d1a8-4ccc-a9b9-79343540f145\") " Oct 30 06:20:02.333703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.333683 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0576b919-84c1-494c-b84a-021af83ccde7-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.333703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.333704 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0576b919-84c1-494c-b84a-021af83ccde7-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.333856 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.333712 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0576b919-84c1-494c-b84a-021af83ccde7-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.333856 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.333720 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0576b919-84c1-494c-b84a-021af83ccde7-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.333856 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.333729 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tpp5\" (UniqueName: \"kubernetes.io/projected/0576b919-84c1-494c-b84a-021af83ccde7-kube-api-access-2tpp5\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.334703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.334658 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-credential-socket" pod "255d0ee6-d1a8-4ccc-a9b9-79343540f145" (UID: "255d0ee6-d1a8-4ccc-a9b9-79343540f145"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.334969 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.334911 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-workload-certs" pod "255d0ee6-d1a8-4ccc-a9b9-79343540f145" (UID: "255d0ee6-d1a8-4ccc-a9b9-79343540f145"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.335119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.335065 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istiod-ca-cert" pod "255d0ee6-d1a8-4ccc-a9b9-79343540f145" (UID: "255d0ee6-d1a8-4ccc-a9b9-79343540f145"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:02.335206 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.335157 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-workload-socket" pod "255d0ee6-d1a8-4ccc-a9b9-79343540f145" (UID: "255d0ee6-d1a8-4ccc-a9b9-79343540f145"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.335309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.335104 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-data" pod "255d0ee6-d1a8-4ccc-a9b9-79343540f145" (UID: "255d0ee6-d1a8-4ccc-a9b9-79343540f145"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.336644 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.336623 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-token" pod "255d0ee6-d1a8-4ccc-a9b9-79343540f145" (UID: "255d0ee6-d1a8-4ccc-a9b9-79343540f145"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.336741 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.336657 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255d0ee6-d1a8-4ccc-a9b9-79343540f145-kube-api-access-lfghp" pod "255d0ee6-d1a8-4ccc-a9b9-79343540f145" (UID: "255d0ee6-d1a8-4ccc-a9b9-79343540f145"). InnerVolumeSpecName "kube-api-access-lfghp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.338179 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.338159 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-envoy" pod "255d0ee6-d1a8-4ccc-a9b9-79343540f145" (UID: "255d0ee6-d1a8-4ccc-a9b9-79343540f145"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.338458 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.338435 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-podinfo" pod "255d0ee6-d1a8-4ccc-a9b9-79343540f145" (UID: "255d0ee6-d1a8-4ccc-a9b9-79343540f145"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:02.435619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.435584 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.435619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.435618 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.435919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.435654 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.435919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.435669 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.435919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.435681 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfghp\" (UniqueName: \"kubernetes.io/projected/255d0ee6-d1a8-4ccc-a9b9-79343540f145-kube-api-access-lfghp\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.435919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.435696 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.435919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.435709 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/255d0ee6-d1a8-4ccc-a9b9-79343540f145-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.435919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.435747 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.435919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.435762 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/255d0ee6-d1a8-4ccc-a9b9-79343540f145-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.546911 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.546880 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:20:02.636021 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.635985 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf"] Oct 30 06:20:02.638018 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.637990 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff75821-8a85-4c01-a62f-f000b1c4d98b-kube-api-access-wrb48\" (UniqueName: \"kubernetes.io/projected/dff75821-8a85-4c01-a62f-f000b1c4d98b-kube-api-access-wrb48\") pod \"dff75821-8a85-4c01-a62f-f000b1c4d98b\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " Oct 30 06:20:02.638172 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.638062 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-workload-certs\") pod \"dff75821-8a85-4c01-a62f-f000b1c4d98b\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " Oct 30 06:20:02.638172 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.638136 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-envoy\") pod \"dff75821-8a85-4c01-a62f-f000b1c4d98b\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " Oct 30 06:20:02.638172 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.638172 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-token\" (UniqueName: \"kubernetes.io/projected/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-token\") pod \"dff75821-8a85-4c01-a62f-f000b1c4d98b\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " Oct 30 06:20:02.638400 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.638252 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-workload-socket\") pod \"dff75821-8a85-4c01-a62f-f000b1c4d98b\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " Oct 30 06:20:02.638400 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.638288 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-podinfo\") pod \"dff75821-8a85-4c01-a62f-f000b1c4d98b\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " Oct 30 06:20:02.638400 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.638341 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-credential-socket\") pod \"dff75821-8a85-4c01-a62f-f000b1c4d98b\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " Oct 30 06:20:02.638610 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.638405 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-data\") pod \"dff75821-8a85-4c01-a62f-f000b1c4d98b\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " Oct 30 06:20:02.638610 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.638426 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff75821-8a85-4c01-a62f-f000b1c4d98b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff75821-8a85-4c01-a62f-f000b1c4d98b-istiod-ca-cert\") pod \"dff75821-8a85-4c01-a62f-f000b1c4d98b\" (UID: \"dff75821-8a85-4c01-a62f-f000b1c4d98b\") " Oct 30 06:20:02.639127 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.639103 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff75821-8a85-4c01-a62f-f000b1c4d98b-istiod-ca-cert" pod "dff75821-8a85-4c01-a62f-f000b1c4d98b" (UID: "dff75821-8a85-4c01-a62f-f000b1c4d98b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:02.639532 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.639510 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-d59rf"] Oct 30 06:20:02.639727 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.639707 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-workload-socket" pod "dff75821-8a85-4c01-a62f-f000b1c4d98b" (UID: "dff75821-8a85-4c01-a62f-f000b1c4d98b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.639890 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.639870 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-workload-certs" pod "dff75821-8a85-4c01-a62f-f000b1c4d98b" (UID: "dff75821-8a85-4c01-a62f-f000b1c4d98b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.640290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.640269 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-credential-socket" pod "dff75821-8a85-4c01-a62f-f000b1c4d98b" (UID: "dff75821-8a85-4c01-a62f-f000b1c4d98b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.640472 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.640451 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-data" pod "dff75821-8a85-4c01-a62f-f000b1c4d98b" (UID: "dff75821-8a85-4c01-a62f-f000b1c4d98b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.641942 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.641915 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff75821-8a85-4c01-a62f-f000b1c4d98b-kube-api-access-wrb48" pod "dff75821-8a85-4c01-a62f-f000b1c4d98b" (UID: "dff75821-8a85-4c01-a62f-f000b1c4d98b"). InnerVolumeSpecName "kube-api-access-wrb48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.642308 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.642290 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-token" pod "dff75821-8a85-4c01-a62f-f000b1c4d98b" (UID: "dff75821-8a85-4c01-a62f-f000b1c4d98b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:02.642379 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.642295 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-podinfo" pod "dff75821-8a85-4c01-a62f-f000b1c4d98b" (UID: "dff75821-8a85-4c01-a62f-f000b1c4d98b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:02.642599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.642580 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-envoy" pod "dff75821-8a85-4c01-a62f-f000b1c4d98b" (UID: "dff75821-8a85-4c01-a62f-f000b1c4d98b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:02.740678 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.740643 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.740678 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.740678 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.740957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.740692 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.740957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.740705 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.740957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.740720 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.740957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.740735 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.740957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.740749 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dff75821-8a85-4c01-a62f-f000b1c4d98b-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.740957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.740781 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dff75821-8a85-4c01-a62f-f000b1c4d98b-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.740957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.740805 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrb48\" (UniqueName: \"kubernetes.io/projected/dff75821-8a85-4c01-a62f-f000b1c4d98b-kube-api-access-wrb48\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:02.900324 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:02.900241 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:03.098409 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:03.098372 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:03.260402 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:03.260323 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" Oct 30 06:20:03.260402 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:03.260350 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc" event={"ID":"dff75821-8a85-4c01-a62f-f000b1c4d98b","Type":"ContainerDied","Data":"c6dc2011549098b2582439f38caca744cabe81408dcf0537b3fb4895f4b0b979"} Oct 30 06:20:03.260928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:03.260412 6999 scope.go:122] "RemoveContainer" containerID="35ff6379fd378a0e52c0e2c4f8920b4e03d4f714858cc68d921d163148a26750" Oct 30 06:20:03.314075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:03.314041 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc"] Oct 30 06:20:03.326670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:03.326645 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0fab6d-openshift-default-78c469c695-8sgwc"] Oct 30 06:20:03.610624 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:03.610597 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0576b919-84c1-494c-b84a-021af83ccde7" path="/var/lib/kubelet/pods/0576b919-84c1-494c-b84a-021af83ccde7/volumes" Oct 30 06:20:03.611042 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:03.611029 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="255d0ee6-d1a8-4ccc-a9b9-79343540f145" path="/var/lib/kubelet/pods/255d0ee6-d1a8-4ccc-a9b9-79343540f145/volumes" Oct 30 06:20:03.611477 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:03.611464 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff75821-8a85-4c01-a62f-f000b1c4d98b" path="/var/lib/kubelet/pods/dff75821-8a85-4c01-a62f-f000b1c4d98b/volumes" Oct 30 06:20:04.141895 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:20:04.141858 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:20:04.450097 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.449974 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf"] Oct 30 06:20:04.450535 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.450205 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" podUID="3786e3f7-54a2-44b9-a7b0-49c5e333f0f0" containerName="maas-api" containerID="cri-o://44336a6ef2dd02899a56a4993373ad53e7c77ea68bb30ee9e8433d6108ee6d6e" gracePeriod=30 Oct 30 06:20:04.548499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.548470 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:04.628481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.628449 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b"] Oct 30 06:20:04.628722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.628697 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" podUID="75d39900-c6a7-48e9-8437-725d2bc7f918" containerName="payload-processing" containerID="cri-o://c6fabe265533b354fa0b504528549c4cb08ac4c53613816019c3abc818fb7b99" gracePeriod=30 Oct 30 06:20:04.654147 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.654118 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf"] Oct 30 06:20:04.658523 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.658382 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" podUID="93c24e78-9db8-4c16-b29a-874b93e894dd" containerName="payload-pre-processing" containerID="cri-o://e2e3b9b522867ab47569c3e852ef55666f610846636fe654f251beb5e41b65ef" gracePeriod=30 Oct 30 06:20:04.745821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.745782 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:04.849583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.849558 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:20:04.886771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.886736 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:04.897425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.896412 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-metrics-certs\") pod \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " Oct 30 06:20:04.897425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.896473 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-kube-api-access-zhvxx\" (UniqueName: \"kubernetes.io/projected/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-kube-api-access-zhvxx\") pod \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " Oct 30 06:20:04.897425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.896502 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-maas-api-tls\") pod \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " Oct 30 06:20:04.897425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.896683 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-openshift-service-ca\") pod \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\" (UID: \"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0\") " Oct 30 06:20:04.901235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.900975 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-openshift-service-ca" pod "3786e3f7-54a2-44b9-a7b0-49c5e333f0f0" (UID: "3786e3f7-54a2-44b9-a7b0-49c5e333f0f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:04.902248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.901416 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-metrics-certs" pod "3786e3f7-54a2-44b9-a7b0-49c5e333f0f0" (UID: "3786e3f7-54a2-44b9-a7b0-49c5e333f0f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:04.904280 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.902719 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-kube-api-access-zhvxx" pod "3786e3f7-54a2-44b9-a7b0-49c5e333f0f0" (UID: "3786e3f7-54a2-44b9-a7b0-49c5e333f0f0"). InnerVolumeSpecName "kube-api-access-zhvxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:04.905082 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.904528 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-maas-api-tls" pod "3786e3f7-54a2-44b9-a7b0-49c5e333f0f0" (UID: "3786e3f7-54a2-44b9-a7b0-49c5e333f0f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:04.972492 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.972407 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" Oct 30 06:20:04.998019 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.997988 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d39900-c6a7-48e9-8437-725d2bc7f918-kube-api-access-4qmg5\" (UniqueName: \"kubernetes.io/projected/75d39900-c6a7-48e9-8437-725d2bc7f918-kube-api-access-4qmg5\") pod \"75d39900-c6a7-48e9-8437-725d2bc7f918\" (UID: \"75d39900-c6a7-48e9-8437-725d2bc7f918\") " Oct 30 06:20:04.998237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.998200 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d39900-c6a7-48e9-8437-725d2bc7f918-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d39900-c6a7-48e9-8437-725d2bc7f918-plugins-config-volume\") pod \"75d39900-c6a7-48e9-8437-725d2bc7f918\" (UID: \"75d39900-c6a7-48e9-8437-725d2bc7f918\") " Oct 30 06:20:04.998540 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.998514 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d39900-c6a7-48e9-8437-725d2bc7f918-plugins-config-volume" pod "75d39900-c6a7-48e9-8437-725d2bc7f918" (UID: "75d39900-c6a7-48e9-8437-725d2bc7f918"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:04.998817 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.998752 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.998817 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.998765 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.998817 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.998774 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhvxx\" (UniqueName: \"kubernetes.io/projected/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-kube-api-access-zhvxx\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.998817 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.998787 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:04.998817 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:04.998798 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75d39900-c6a7-48e9-8437-725d2bc7f918-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:05.000449 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.000424 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d39900-c6a7-48e9-8437-725d2bc7f918-kube-api-access-4qmg5" pod "75d39900-c6a7-48e9-8437-725d2bc7f918" (UID: "75d39900-c6a7-48e9-8437-725d2bc7f918"). InnerVolumeSpecName "kube-api-access-4qmg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:05.010204 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.010180 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" Oct 30 06:20:05.102404 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.102364 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qmg5\" (UniqueName: \"kubernetes.io/projected/75d39900-c6a7-48e9-8437-725d2bc7f918-kube-api-access-4qmg5\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:05.177510 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.177472 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:05.203564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.203532 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93c24e78-9db8-4c16-b29a-874b93e894dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c24e78-9db8-4c16-b29a-874b93e894dd-plugins-config-volume\") pod \"93c24e78-9db8-4c16-b29a-874b93e894dd\" (UID: \"93c24e78-9db8-4c16-b29a-874b93e894dd\") " Oct 30 06:20:05.203712 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.203595 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93c24e78-9db8-4c16-b29a-874b93e894dd-kube-api-access-wr7bj\" (UniqueName: \"kubernetes.io/projected/93c24e78-9db8-4c16-b29a-874b93e894dd-kube-api-access-wr7bj\") pod \"93c24e78-9db8-4c16-b29a-874b93e894dd\" (UID: \"93c24e78-9db8-4c16-b29a-874b93e894dd\") " Oct 30 06:20:05.203885 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.203865 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93c24e78-9db8-4c16-b29a-874b93e894dd-plugins-config-volume" pod "93c24e78-9db8-4c16-b29a-874b93e894dd" (UID: "93c24e78-9db8-4c16-b29a-874b93e894dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:05.204128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.204110 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93c24e78-9db8-4c16-b29a-874b93e894dd-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:05.205745 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.205725 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93c24e78-9db8-4c16-b29a-874b93e894dd-kube-api-access-wr7bj" pod "93c24e78-9db8-4c16-b29a-874b93e894dd" (UID: "93c24e78-9db8-4c16-b29a-874b93e894dd"). InnerVolumeSpecName "kube-api-access-wr7bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:05.286605 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.286508 6999 generic.go:356] "Generic (PLEG): container finished" podID="3786e3f7-54a2-44b9-a7b0-49c5e333f0f0" containerID="44336a6ef2dd02899a56a4993373ad53e7c77ea68bb30ee9e8433d6108ee6d6e" exitCode=0 Oct 30 06:20:05.286605 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.286580 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" Oct 30 06:20:05.286812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.286596 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" event={"ID":"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0","Type":"ContainerDied","Data":"44336a6ef2dd02899a56a4993373ad53e7c77ea68bb30ee9e8433d6108ee6d6e"} Oct 30 06:20:05.286812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.286681 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf" event={"ID":"3786e3f7-54a2-44b9-a7b0-49c5e333f0f0","Type":"ContainerDied","Data":"0e1bfbff14db0213dda2844da5b9b0c25cb78bb95cbdd9298169b7f08156de86"} Oct 30 06:20:05.286812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.286716 6999 scope.go:122] "RemoveContainer" containerID="44336a6ef2dd02899a56a4993373ad53e7c77ea68bb30ee9e8433d6108ee6d6e" Oct 30 06:20:05.287977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.287952 6999 generic.go:356] "Generic (PLEG): container finished" podID="93c24e78-9db8-4c16-b29a-874b93e894dd" containerID="e2e3b9b522867ab47569c3e852ef55666f610846636fe654f251beb5e41b65ef" exitCode=0 Oct 30 06:20:05.288087 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.288022 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" Oct 30 06:20:05.288087 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.288038 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" event={"ID":"93c24e78-9db8-4c16-b29a-874b93e894dd","Type":"ContainerDied","Data":"e2e3b9b522867ab47569c3e852ef55666f610846636fe654f251beb5e41b65ef"} Oct 30 06:20:05.288191 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.288092 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf" event={"ID":"93c24e78-9db8-4c16-b29a-874b93e894dd","Type":"ContainerDied","Data":"05300a8989cc7fb7dd9aaa0ddee8dd4fa0c3897a7dd9de95cd1252cc4e11cbfc"} Oct 30 06:20:05.289677 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.289653 6999 generic.go:356] "Generic (PLEG): container finished" podID="75d39900-c6a7-48e9-8437-725d2bc7f918" containerID="c6fabe265533b354fa0b504528549c4cb08ac4c53613816019c3abc818fb7b99" exitCode=0 Oct 30 06:20:05.289805 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.289717 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" Oct 30 06:20:05.289805 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.289723 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" event={"ID":"75d39900-c6a7-48e9-8437-725d2bc7f918","Type":"ContainerDied","Data":"c6fabe265533b354fa0b504528549c4cb08ac4c53613816019c3abc818fb7b99"} Oct 30 06:20:05.289805 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.289751 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b" event={"ID":"75d39900-c6a7-48e9-8437-725d2bc7f918","Type":"ContainerDied","Data":"f38fc2ec439ad0c0fe0030a5acb1a628780227dd0dd75bf5c70bad9691a84680"} Oct 30 06:20:05.299058 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.299031 6999 scope.go:122] "RemoveContainer" containerID="44336a6ef2dd02899a56a4993373ad53e7c77ea68bb30ee9e8433d6108ee6d6e" Oct 30 06:20:05.299388 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:20:05.299367 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44336a6ef2dd02899a56a4993373ad53e7c77ea68bb30ee9e8433d6108ee6d6e\": container with ID starting with 44336a6ef2dd02899a56a4993373ad53e7c77ea68bb30ee9e8433d6108ee6d6e not found: ID does not exist" containerID="44336a6ef2dd02899a56a4993373ad53e7c77ea68bb30ee9e8433d6108ee6d6e" Oct 30 06:20:05.299496 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.299394 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44336a6ef2dd02899a56a4993373ad53e7c77ea68bb30ee9e8433d6108ee6d6e"} err="failed to get container status \"44336a6ef2dd02899a56a4993373ad53e7c77ea68bb30ee9e8433d6108ee6d6e\": rpc error: code = NotFound desc = could not find container \"44336a6ef2dd02899a56a4993373ad53e7c77ea68bb30ee9e8433d6108ee6d6e\": container with ID starting with 44336a6ef2dd02899a56a4993373ad53e7c77ea68bb30ee9e8433d6108ee6d6e not found: ID does not exist" Oct 30 06:20:05.299496 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.299412 6999 scope.go:122] "RemoveContainer" containerID="e2e3b9b522867ab47569c3e852ef55666f610846636fe654f251beb5e41b65ef" Oct 30 06:20:05.305689 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.305664 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr7bj\" (UniqueName: \"kubernetes.io/projected/93c24e78-9db8-4c16-b29a-874b93e894dd-kube-api-access-wr7bj\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:05.308889 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.308868 6999 scope.go:122] "RemoveContainer" containerID="e2e3b9b522867ab47569c3e852ef55666f610846636fe654f251beb5e41b65ef" Oct 30 06:20:05.309166 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:20:05.309148 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e3b9b522867ab47569c3e852ef55666f610846636fe654f251beb5e41b65ef\": container with ID starting with e2e3b9b522867ab47569c3e852ef55666f610846636fe654f251beb5e41b65ef not found: ID does not exist" containerID="e2e3b9b522867ab47569c3e852ef55666f610846636fe654f251beb5e41b65ef" Oct 30 06:20:05.309259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.309174 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e3b9b522867ab47569c3e852ef55666f610846636fe654f251beb5e41b65ef"} err="failed to get container status \"e2e3b9b522867ab47569c3e852ef55666f610846636fe654f251beb5e41b65ef\": rpc error: code = NotFound desc = could not find container \"e2e3b9b522867ab47569c3e852ef55666f610846636fe654f251beb5e41b65ef\": container with ID starting with e2e3b9b522867ab47569c3e852ef55666f610846636fe654f251beb5e41b65ef not found: ID does not exist" Oct 30 06:20:05.309259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.309190 6999 scope.go:122] "RemoveContainer" containerID="c6fabe265533b354fa0b504528549c4cb08ac4c53613816019c3abc818fb7b99" Oct 30 06:20:05.318373 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.318355 6999 scope.go:122] "RemoveContainer" containerID="c6fabe265533b354fa0b504528549c4cb08ac4c53613816019c3abc818fb7b99" Oct 30 06:20:05.318631 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:20:05.318613 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6fabe265533b354fa0b504528549c4cb08ac4c53613816019c3abc818fb7b99\": container with ID starting with c6fabe265533b354fa0b504528549c4cb08ac4c53613816019c3abc818fb7b99 not found: ID does not exist" containerID="c6fabe265533b354fa0b504528549c4cb08ac4c53613816019c3abc818fb7b99" Oct 30 06:20:05.318689 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.318639 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6fabe265533b354fa0b504528549c4cb08ac4c53613816019c3abc818fb7b99"} err="failed to get container status \"c6fabe265533b354fa0b504528549c4cb08ac4c53613816019c3abc818fb7b99\": rpc error: code = NotFound desc = could not find container \"c6fabe265533b354fa0b504528549c4cb08ac4c53613816019c3abc818fb7b99\": container with ID starting with c6fabe265533b354fa0b504528549c4cb08ac4c53613816019c3abc818fb7b99 not found: ID does not exist" Oct 30 06:20:05.358636 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.358608 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf"] Oct 30 06:20:05.377714 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.377673 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-584c9d-54674c7bd4-fddvf"] Oct 30 06:20:05.404294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.404256 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf"] Oct 30 06:20:05.408700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.408675 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-584c9d-7cf7cb4b67-qhcgf"] Oct 30 06:20:05.441917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.441889 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b"] Oct 30 06:20:05.444195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.444126 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:05.446950 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.446930 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-584c9d-5947df7d8-g6p5b"] Oct 30 06:20:05.585994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.585920 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:05.610095 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.610067 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3786e3f7-54a2-44b9-a7b0-49c5e333f0f0" path="/var/lib/kubelet/pods/3786e3f7-54a2-44b9-a7b0-49c5e333f0f0/volumes" Oct 30 06:20:05.610499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.610484 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d39900-c6a7-48e9-8437-725d2bc7f918" path="/var/lib/kubelet/pods/75d39900-c6a7-48e9-8437-725d2bc7f918/volumes" Oct 30 06:20:05.610828 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.610816 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93c24e78-9db8-4c16-b29a-874b93e894dd" path="/var/lib/kubelet/pods/93c24e78-9db8-4c16-b29a-874b93e894dd/volumes" Oct 30 06:20:05.949406 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:05.949375 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:07.386784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:07.386752 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:07.948467 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:07.948433 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd"] Oct 30 06:20:08.086839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:08.086810 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:08.223082 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:08.223003 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:09.358325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:09.358290 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:11.775592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:11.775557 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:12.584443 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:12.584411 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:12.717993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:12.717944 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:12.949396 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:12.949358 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" podUID="47351595-ca08-49ed-bd29-d8046678774a" containerName="payload-pre-processing" containerID="cri-o://87560a0f159be7b7cd585af21bfc8b487e323bd51fa7d7e957fd422d9a88a3c1" gracePeriod=25 Oct 30 06:20:12.952647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:12.952617 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:13.100611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.100584 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:13.254835 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.254785 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:13.331508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.331480 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" Oct 30 06:20:13.346666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.346641 6999 generic.go:356] "Generic (PLEG): container finished" podID="47351595-ca08-49ed-bd29-d8046678774a" containerID="87560a0f159be7b7cd585af21bfc8b487e323bd51fa7d7e957fd422d9a88a3c1" exitCode=0 Oct 30 06:20:13.346808 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.346677 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" event={"ID":"47351595-ca08-49ed-bd29-d8046678774a","Type":"ContainerDied","Data":"87560a0f159be7b7cd585af21bfc8b487e323bd51fa7d7e957fd422d9a88a3c1"} Oct 30 06:20:13.346808 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.346680 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" Oct 30 06:20:13.346808 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.346715 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd" event={"ID":"47351595-ca08-49ed-bd29-d8046678774a","Type":"ContainerDied","Data":"eede63a0b23cb9b5bb11299844744bdcdf83bf015350fe7df81c92bbdfec497f"} Oct 30 06:20:13.346808 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.346736 6999 scope.go:122] "RemoveContainer" containerID="87560a0f159be7b7cd585af21bfc8b487e323bd51fa7d7e957fd422d9a88a3c1" Oct 30 06:20:13.356785 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.356759 6999 scope.go:122] "RemoveContainer" containerID="87560a0f159be7b7cd585af21bfc8b487e323bd51fa7d7e957fd422d9a88a3c1" Oct 30 06:20:13.357048 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:20:13.357032 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87560a0f159be7b7cd585af21bfc8b487e323bd51fa7d7e957fd422d9a88a3c1\": container with ID starting with 87560a0f159be7b7cd585af21bfc8b487e323bd51fa7d7e957fd422d9a88a3c1 not found: ID does not exist" containerID="87560a0f159be7b7cd585af21bfc8b487e323bd51fa7d7e957fd422d9a88a3c1" Oct 30 06:20:13.357099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.357058 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87560a0f159be7b7cd585af21bfc8b487e323bd51fa7d7e957fd422d9a88a3c1"} err="failed to get container status \"87560a0f159be7b7cd585af21bfc8b487e323bd51fa7d7e957fd422d9a88a3c1\": rpc error: code = NotFound desc = could not find container \"87560a0f159be7b7cd585af21bfc8b487e323bd51fa7d7e957fd422d9a88a3c1\": container with ID starting with 87560a0f159be7b7cd585af21bfc8b487e323bd51fa7d7e957fd422d9a88a3c1 not found: ID does not exist" Oct 30 06:20:13.433069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.432193 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47351595-ca08-49ed-bd29-d8046678774a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47351595-ca08-49ed-bd29-d8046678774a-plugins-config-volume\") pod \"47351595-ca08-49ed-bd29-d8046678774a\" (UID: \"47351595-ca08-49ed-bd29-d8046678774a\") " Oct 30 06:20:13.433069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.432451 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47351595-ca08-49ed-bd29-d8046678774a-kube-api-access-cwz5v\" (UniqueName: \"kubernetes.io/projected/47351595-ca08-49ed-bd29-d8046678774a-kube-api-access-cwz5v\") pod \"47351595-ca08-49ed-bd29-d8046678774a\" (UID: \"47351595-ca08-49ed-bd29-d8046678774a\") " Oct 30 06:20:13.433069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.432757 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47351595-ca08-49ed-bd29-d8046678774a-plugins-config-volume" pod "47351595-ca08-49ed-bd29-d8046678774a" (UID: "47351595-ca08-49ed-bd29-d8046678774a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:13.433069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.433019 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47351595-ca08-49ed-bd29-d8046678774a-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:13.434957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.434931 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47351595-ca08-49ed-bd29-d8046678774a-kube-api-access-cwz5v" pod "47351595-ca08-49ed-bd29-d8046678774a" (UID: "47351595-ca08-49ed-bd29-d8046678774a"). InnerVolumeSpecName "kube-api-access-cwz5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:13.534681 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.534589 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwz5v\" (UniqueName: \"kubernetes.io/projected/47351595-ca08-49ed-bd29-d8046678774a-kube-api-access-cwz5v\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:13.687932 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.687906 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd"] Oct 30 06:20:13.690312 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:13.690288 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31407e-5969d8795-tl5rd"] Oct 30 06:20:14.268686 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:14.268659 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:15.609532 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:15.609506 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47351595-ca08-49ed-bd29-d8046678774a" path="/var/lib/kubelet/pods/47351595-ca08-49ed-bd29-d8046678774a/volumes" Oct 30 06:20:16.275434 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:16.275399 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:18.663837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:18.663803 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:20.418060 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:20.418019 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:21.820040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:21.820005 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh"] Oct 30 06:20:21.820502 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:21.820278 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" podUID="ace3fa7d-d193-4184-8850-c4741edd37b3" containerName="maas-api" containerID="cri-o://62c2316cbe7e39f7ac0b2e8ab72f2a49ce620d0611e73e31641449241fd1ac8c" gracePeriod=30 Oct 30 06:20:21.909186 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:21.909121 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:21.999155 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:21.999128 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf"] Oct 30 06:20:21.999407 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:21.999381 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" podUID="c62893d3-85ac-42d9-a5f4-e7eba0aa5924" containerName="payload-processing" containerID="cri-o://9b5f96fb0b44c5469f258edf914b662dec7c36317e3317104dcbe8b66c5520d9" gracePeriod=30 Oct 30 06:20:22.021289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.021262 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d"] Oct 30 06:20:22.021725 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.021676 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" podUID="8f277c85-c9a4-47ed-9c18-96fcdf78728e" containerName="payload-pre-processing" containerID="cri-o://c709d5cc48deb96b2f6c2c4055c3dbab5c54ccf1d594be4820f555e2bfdab962" gracePeriod=30 Oct 30 06:20:22.130712 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.130669 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:22.178573 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.178544 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:20:22.270325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.270295 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:22.331498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.331472 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" Oct 30 06:20:22.372196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.363078 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace3fa7d-d193-4184-8850-c4741edd37b3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace3fa7d-d193-4184-8850-c4741edd37b3-openshift-service-ca\") pod \"ace3fa7d-d193-4184-8850-c4741edd37b3\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " Oct 30 06:20:22.372196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.363126 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-maas-api-tls\") pod \"ace3fa7d-d193-4184-8850-c4741edd37b3\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " Oct 30 06:20:22.372196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.363337 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace3fa7d-d193-4184-8850-c4741edd37b3-kube-api-access-ngpgh\" (UniqueName: \"kubernetes.io/projected/ace3fa7d-d193-4184-8850-c4741edd37b3-kube-api-access-ngpgh\") pod \"ace3fa7d-d193-4184-8850-c4741edd37b3\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " Oct 30 06:20:22.372196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.363367 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-metrics-certs\") pod \"ace3fa7d-d193-4184-8850-c4741edd37b3\" (UID: \"ace3fa7d-d193-4184-8850-c4741edd37b3\") " Oct 30 06:20:22.398963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.379456 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace3fa7d-d193-4184-8850-c4741edd37b3-openshift-service-ca" pod "ace3fa7d-d193-4184-8850-c4741edd37b3" (UID: "ace3fa7d-d193-4184-8850-c4741edd37b3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:22.407697 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.406553 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-maas-api-tls" pod "ace3fa7d-d193-4184-8850-c4741edd37b3" (UID: "ace3fa7d-d193-4184-8850-c4741edd37b3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:22.450854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.450696 6999 generic.go:356] "Generic (PLEG): container finished" podID="c62893d3-85ac-42d9-a5f4-e7eba0aa5924" containerID="9b5f96fb0b44c5469f258edf914b662dec7c36317e3317104dcbe8b66c5520d9" exitCode=0 Oct 30 06:20:22.451766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.451085 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" event={"ID":"c62893d3-85ac-42d9-a5f4-e7eba0aa5924","Type":"ContainerDied","Data":"9b5f96fb0b44c5469f258edf914b662dec7c36317e3317104dcbe8b66c5520d9"} Oct 30 06:20:22.451766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.451156 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" event={"ID":"c62893d3-85ac-42d9-a5f4-e7eba0aa5924","Type":"ContainerDied","Data":"47306fefeaa991f8307f5018a9832f0ca5d9987b19f0a169853abadadf0b51dd"} Oct 30 06:20:22.451766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.451228 6999 scope.go:122] "RemoveContainer" containerID="9b5f96fb0b44c5469f258edf914b662dec7c36317e3317104dcbe8b66c5520d9" Oct 30 06:20:22.451766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.451409 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf" Oct 30 06:20:22.455828 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.454436 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-metrics-certs" pod "ace3fa7d-d193-4184-8850-c4741edd37b3" (UID: "ace3fa7d-d193-4184-8850-c4741edd37b3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:22.466302 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.465860 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c62893d3-85ac-42d9-a5f4-e7eba0aa5924-kube-api-access-fzpxc\" (UniqueName: \"kubernetes.io/projected/c62893d3-85ac-42d9-a5f4-e7eba0aa5924-kube-api-access-fzpxc\") pod \"c62893d3-85ac-42d9-a5f4-e7eba0aa5924\" (UID: \"c62893d3-85ac-42d9-a5f4-e7eba0aa5924\") " Oct 30 06:20:22.466302 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.465937 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c62893d3-85ac-42d9-a5f4-e7eba0aa5924-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62893d3-85ac-42d9-a5f4-e7eba0aa5924-plugins-config-volume\") pod \"c62893d3-85ac-42d9-a5f4-e7eba0aa5924\" (UID: \"c62893d3-85ac-42d9-a5f4-e7eba0aa5924\") " Oct 30 06:20:22.471385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.466541 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:22.471385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.466566 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ace3fa7d-d193-4184-8850-c4741edd37b3-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:22.471385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.466584 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ace3fa7d-d193-4184-8850-c4741edd37b3-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:22.471385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.468852 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace3fa7d-d193-4184-8850-c4741edd37b3-kube-api-access-ngpgh" pod "ace3fa7d-d193-4184-8850-c4741edd37b3" (UID: "ace3fa7d-d193-4184-8850-c4741edd37b3"). InnerVolumeSpecName "kube-api-access-ngpgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:22.471385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.470461 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c62893d3-85ac-42d9-a5f4-e7eba0aa5924-plugins-config-volume" pod "c62893d3-85ac-42d9-a5f4-e7eba0aa5924" (UID: "c62893d3-85ac-42d9-a5f4-e7eba0aa5924"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:22.486176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.485153 6999 generic.go:356] "Generic (PLEG): container finished" podID="8f277c85-c9a4-47ed-9c18-96fcdf78728e" containerID="c709d5cc48deb96b2f6c2c4055c3dbab5c54ccf1d594be4820f555e2bfdab962" exitCode=0 Oct 30 06:20:22.486176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.485368 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" event={"ID":"8f277c85-c9a4-47ed-9c18-96fcdf78728e","Type":"ContainerDied","Data":"c709d5cc48deb96b2f6c2c4055c3dbab5c54ccf1d594be4820f555e2bfdab962"} Oct 30 06:20:22.489344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.489023 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" Oct 30 06:20:22.494119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.494072 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62893d3-85ac-42d9-a5f4-e7eba0aa5924-kube-api-access-fzpxc" pod "c62893d3-85ac-42d9-a5f4-e7eba0aa5924" (UID: "c62893d3-85ac-42d9-a5f4-e7eba0aa5924"). InnerVolumeSpecName "kube-api-access-fzpxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:22.495995 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.495242 6999 generic.go:356] "Generic (PLEG): container finished" podID="ace3fa7d-d193-4184-8850-c4741edd37b3" containerID="62c2316cbe7e39f7ac0b2e8ab72f2a49ce620d0611e73e31641449241fd1ac8c" exitCode=0 Oct 30 06:20:22.495995 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.495294 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" event={"ID":"ace3fa7d-d193-4184-8850-c4741edd37b3","Type":"ContainerDied","Data":"62c2316cbe7e39f7ac0b2e8ab72f2a49ce620d0611e73e31641449241fd1ac8c"} Oct 30 06:20:22.495995 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.495329 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" event={"ID":"ace3fa7d-d193-4184-8850-c4741edd37b3","Type":"ContainerDied","Data":"0ac71a62e4b628ae9c8ad854f60a73a2e038b01982166ac00f52151fee85f393"} Oct 30 06:20:22.496580 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.496197 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh" Oct 30 06:20:22.500961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.500944 6999 scope.go:122] "RemoveContainer" containerID="9b5f96fb0b44c5469f258edf914b662dec7c36317e3317104dcbe8b66c5520d9" Oct 30 06:20:22.501282 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:20:22.501260 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b5f96fb0b44c5469f258edf914b662dec7c36317e3317104dcbe8b66c5520d9\": container with ID starting with 9b5f96fb0b44c5469f258edf914b662dec7c36317e3317104dcbe8b66c5520d9 not found: ID does not exist" containerID="9b5f96fb0b44c5469f258edf914b662dec7c36317e3317104dcbe8b66c5520d9" Oct 30 06:20:22.501376 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.501295 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b5f96fb0b44c5469f258edf914b662dec7c36317e3317104dcbe8b66c5520d9"} err="failed to get container status \"9b5f96fb0b44c5469f258edf914b662dec7c36317e3317104dcbe8b66c5520d9\": rpc error: code = NotFound desc = could not find container \"9b5f96fb0b44c5469f258edf914b662dec7c36317e3317104dcbe8b66c5520d9\": container with ID starting with 9b5f96fb0b44c5469f258edf914b662dec7c36317e3317104dcbe8b66c5520d9 not found: ID does not exist" Oct 30 06:20:22.501376 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.501323 6999 scope.go:122] "RemoveContainer" containerID="62c2316cbe7e39f7ac0b2e8ab72f2a49ce620d0611e73e31641449241fd1ac8c" Oct 30 06:20:22.515916 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.515898 6999 scope.go:122] "RemoveContainer" containerID="62c2316cbe7e39f7ac0b2e8ab72f2a49ce620d0611e73e31641449241fd1ac8c" Oct 30 06:20:22.516288 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:20:22.516266 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c2316cbe7e39f7ac0b2e8ab72f2a49ce620d0611e73e31641449241fd1ac8c\": container with ID starting with 62c2316cbe7e39f7ac0b2e8ab72f2a49ce620d0611e73e31641449241fd1ac8c not found: ID does not exist" containerID="62c2316cbe7e39f7ac0b2e8ab72f2a49ce620d0611e73e31641449241fd1ac8c" Oct 30 06:20:22.516376 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.516298 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c2316cbe7e39f7ac0b2e8ab72f2a49ce620d0611e73e31641449241fd1ac8c"} err="failed to get container status \"62c2316cbe7e39f7ac0b2e8ab72f2a49ce620d0611e73e31641449241fd1ac8c\": rpc error: code = NotFound desc = could not find container \"62c2316cbe7e39f7ac0b2e8ab72f2a49ce620d0611e73e31641449241fd1ac8c\": container with ID starting with 62c2316cbe7e39f7ac0b2e8ab72f2a49ce620d0611e73e31641449241fd1ac8c not found: ID does not exist" Oct 30 06:20:22.568075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.567368 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f277c85-c9a4-47ed-9c18-96fcdf78728e-kube-api-access-hqvmj\" (UniqueName: \"kubernetes.io/projected/8f277c85-c9a4-47ed-9c18-96fcdf78728e-kube-api-access-hqvmj\") pod \"8f277c85-c9a4-47ed-9c18-96fcdf78728e\" (UID: \"8f277c85-c9a4-47ed-9c18-96fcdf78728e\") " Oct 30 06:20:22.568075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.567401 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh"] Oct 30 06:20:22.568075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.567511 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f277c85-c9a4-47ed-9c18-96fcdf78728e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f277c85-c9a4-47ed-9c18-96fcdf78728e-plugins-config-volume\") pod \"8f277c85-c9a4-47ed-9c18-96fcdf78728e\" (UID: \"8f277c85-c9a4-47ed-9c18-96fcdf78728e\") " Oct 30 06:20:22.568075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.567937 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f277c85-c9a4-47ed-9c18-96fcdf78728e-plugins-config-volume" pod "8f277c85-c9a4-47ed-9c18-96fcdf78728e" (UID: "8f277c85-c9a4-47ed-9c18-96fcdf78728e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:22.568075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.568001 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzpxc\" (UniqueName: \"kubernetes.io/projected/c62893d3-85ac-42d9-a5f4-e7eba0aa5924-kube-api-access-fzpxc\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:22.568075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.568019 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c62893d3-85ac-42d9-a5f4-e7eba0aa5924-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:22.568075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.568032 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngpgh\" (UniqueName: \"kubernetes.io/projected/ace3fa7d-d193-4184-8850-c4741edd37b3-kube-api-access-ngpgh\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:22.570115 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.570079 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f277c85-c9a4-47ed-9c18-96fcdf78728e-kube-api-access-hqvmj" pod "8f277c85-c9a4-47ed-9c18-96fcdf78728e" (UID: "8f277c85-c9a4-47ed-9c18-96fcdf78728e"). InnerVolumeSpecName "kube-api-access-hqvmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:22.573499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.573477 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-33c99c-56bd64b8b7-wptwh"] Oct 30 06:20:22.594750 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.594729 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:22.669245 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.669142 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqvmj\" (UniqueName: \"kubernetes.io/projected/8f277c85-c9a4-47ed-9c18-96fcdf78728e-kube-api-access-hqvmj\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:22.669245 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.669169 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f277c85-c9a4-47ed-9c18-96fcdf78728e-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:22.738982 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.738948 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:22.797168 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.797142 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf"] Oct 30 06:20:22.802315 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.802287 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-33c99c-5f5c495f66-hmmgf"] Oct 30 06:20:22.892797 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:22.892768 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:23.246641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:23.246612 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:23.504102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:23.504020 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" Oct 30 06:20:23.504102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:23.504054 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d" event={"ID":"8f277c85-c9a4-47ed-9c18-96fcdf78728e","Type":"ContainerDied","Data":"233bbf3814c689e5d4170f8b576112d0afbf541cc6abcaf3601b9971bf8bccd7"} Oct 30 06:20:23.504361 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:23.504127 6999 scope.go:122] "RemoveContainer" containerID="c709d5cc48deb96b2f6c2c4055c3dbab5c54ccf1d594be4820f555e2bfdab962" Oct 30 06:20:23.555906 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:23.555877 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d"] Oct 30 06:20:23.568151 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:23.568128 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-33c99c-74989c67f-hs88d"] Oct 30 06:20:23.608773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:23.608743 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f277c85-c9a4-47ed-9c18-96fcdf78728e" path="/var/lib/kubelet/pods/8f277c85-c9a4-47ed-9c18-96fcdf78728e/volumes" Oct 30 06:20:23.609267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:23.609251 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace3fa7d-d193-4184-8850-c4741edd37b3" path="/var/lib/kubelet/pods/ace3fa7d-d193-4184-8850-c4741edd37b3/volumes" Oct 30 06:20:23.609617 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:23.609604 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c62893d3-85ac-42d9-a5f4-e7eba0aa5924" path="/var/lib/kubelet/pods/c62893d3-85ac-42d9-a5f4-e7eba0aa5924/volumes" Oct 30 06:20:25.367425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:25.367384 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:26.834299 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:26.834271 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll"] Oct 30 06:20:26.834778 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:26.834470 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" podUID="928adf71-31d6-41fa-88eb-159cc0545a8c" containerName="maas-api" containerID="cri-o://5bfd740d7fa3be4e068de75c63ae5efd60e37fe37beacde5c4fc3a2e38c8e750" gracePeriod=30 Oct 30 06:20:27.082424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.082389 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:27.152242 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.150405 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:20:27.250835 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.250808 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:27.282960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.282933 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/928adf71-31d6-41fa-88eb-159cc0545a8c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/928adf71-31d6-41fa-88eb-159cc0545a8c-openshift-service-ca\") pod \"928adf71-31d6-41fa-88eb-159cc0545a8c\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " Oct 30 06:20:27.283098 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.283057 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928adf71-31d6-41fa-88eb-159cc0545a8c-kube-api-access-gk24r\" (UniqueName: \"kubernetes.io/projected/928adf71-31d6-41fa-88eb-159cc0545a8c-kube-api-access-gk24r\") pod \"928adf71-31d6-41fa-88eb-159cc0545a8c\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " Oct 30 06:20:27.283208 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.283184 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/928adf71-31d6-41fa-88eb-159cc0545a8c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928adf71-31d6-41fa-88eb-159cc0545a8c-maas-api-tls\") pod \"928adf71-31d6-41fa-88eb-159cc0545a8c\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " Oct 30 06:20:27.283332 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.283316 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/928adf71-31d6-41fa-88eb-159cc0545a8c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/928adf71-31d6-41fa-88eb-159cc0545a8c-metrics-certs\") pod \"928adf71-31d6-41fa-88eb-159cc0545a8c\" (UID: \"928adf71-31d6-41fa-88eb-159cc0545a8c\") " Oct 30 06:20:27.283416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.283397 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/928adf71-31d6-41fa-88eb-159cc0545a8c-openshift-service-ca" pod "928adf71-31d6-41fa-88eb-159cc0545a8c" (UID: "928adf71-31d6-41fa-88eb-159cc0545a8c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:27.283872 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.283803 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/928adf71-31d6-41fa-88eb-159cc0545a8c-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:27.285277 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.285246 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/928adf71-31d6-41fa-88eb-159cc0545a8c-maas-api-tls" pod "928adf71-31d6-41fa-88eb-159cc0545a8c" (UID: "928adf71-31d6-41fa-88eb-159cc0545a8c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:27.285379 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.285343 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928adf71-31d6-41fa-88eb-159cc0545a8c-kube-api-access-gk24r" pod "928adf71-31d6-41fa-88eb-159cc0545a8c" (UID: "928adf71-31d6-41fa-88eb-159cc0545a8c"). InnerVolumeSpecName "kube-api-access-gk24r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:27.285379 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.285361 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/928adf71-31d6-41fa-88eb-159cc0545a8c-metrics-certs" pod "928adf71-31d6-41fa-88eb-159cc0545a8c" (UID: "928adf71-31d6-41fa-88eb-159cc0545a8c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:27.385192 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.385133 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/928adf71-31d6-41fa-88eb-159cc0545a8c-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:27.385192 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.385154 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/928adf71-31d6-41fa-88eb-159cc0545a8c-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:27.385192 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.385164 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk24r\" (UniqueName: \"kubernetes.io/projected/928adf71-31d6-41fa-88eb-159cc0545a8c-kube-api-access-gk24r\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:27.484015 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.483976 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:27.534353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.534311 6999 generic.go:356] "Generic (PLEG): container finished" podID="928adf71-31d6-41fa-88eb-159cc0545a8c" containerID="5bfd740d7fa3be4e068de75c63ae5efd60e37fe37beacde5c4fc3a2e38c8e750" exitCode=0 Oct 30 06:20:27.534566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.534382 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" event={"ID":"928adf71-31d6-41fa-88eb-159cc0545a8c","Type":"ContainerDied","Data":"5bfd740d7fa3be4e068de75c63ae5efd60e37fe37beacde5c4fc3a2e38c8e750"} Oct 30 06:20:27.534566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.534443 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" Oct 30 06:20:27.534566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.534450 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll" event={"ID":"928adf71-31d6-41fa-88eb-159cc0545a8c","Type":"ContainerDied","Data":"4e857fbb83ed9a6803863736086fdb422391eeabbe3fda42feb998d67f856f27"} Oct 30 06:20:27.534566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.534479 6999 scope.go:122] "RemoveContainer" containerID="5bfd740d7fa3be4e068de75c63ae5efd60e37fe37beacde5c4fc3a2e38c8e750" Oct 30 06:20:27.546054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.545724 6999 scope.go:122] "RemoveContainer" containerID="5bfd740d7fa3be4e068de75c63ae5efd60e37fe37beacde5c4fc3a2e38c8e750" Oct 30 06:20:27.546176 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:20:27.546123 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bfd740d7fa3be4e068de75c63ae5efd60e37fe37beacde5c4fc3a2e38c8e750\": container with ID starting with 5bfd740d7fa3be4e068de75c63ae5efd60e37fe37beacde5c4fc3a2e38c8e750 not found: ID does not exist" containerID="5bfd740d7fa3be4e068de75c63ae5efd60e37fe37beacde5c4fc3a2e38c8e750" Oct 30 06:20:27.546176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.546150 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bfd740d7fa3be4e068de75c63ae5efd60e37fe37beacde5c4fc3a2e38c8e750"} err="failed to get container status \"5bfd740d7fa3be4e068de75c63ae5efd60e37fe37beacde5c4fc3a2e38c8e750\": rpc error: code = NotFound desc = could not find container \"5bfd740d7fa3be4e068de75c63ae5efd60e37fe37beacde5c4fc3a2e38c8e750\": container with ID starting with 5bfd740d7fa3be4e068de75c63ae5efd60e37fe37beacde5c4fc3a2e38c8e750 not found: ID does not exist" Oct 30 06:20:27.592541 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.592510 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll"] Oct 30 06:20:27.614628 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.614593 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8272af-7c75496b94-cmxll"] Oct 30 06:20:27.673578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.673496 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:27.821782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:27.821753 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:28.349395 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:28.349360 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:29.608989 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:29.608956 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="928adf71-31d6-41fa-88eb-159cc0545a8c" path="/var/lib/kubelet/pods/928adf71-31d6-41fa-88eb-159cc0545a8c/volumes" Oct 30 06:20:29.846355 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:29.846323 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:30.833098 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:30.833066 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:30.975386 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:30.975357 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:31.193513 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:31.193485 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:31.721520 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:31.721491 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:33.591803 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:33.591771 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:34.142201 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:20:34.142162 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:20:34.325677 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:34.325641 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:34.476121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:34.476029 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:34.621915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:34.621886 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:35.769483 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:35.769453 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:35.915523 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:35.915494 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:36.077817 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:36.077726 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:36.245316 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:36.245282 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:37.916128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:37.916098 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:38.431661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:38.431623 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:38.622234 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:38.619900 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:38.796473 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:38.796350 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:40.322427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.322395 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97"] Oct 30 06:20:40.322899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.322631 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" podUID="e010dc1f-4cce-448d-bfea-0e499f8c03a4" containerName="maas-api" containerID="cri-o://2139307acccc2484cccb6cd24304dc88cdaa7851e59e2c50fbb8d4177cb1a4a2" gracePeriod=30 Oct 30 06:20:40.419205 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.419154 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:40.491516 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.491471 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv"] Oct 30 06:20:40.491745 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.491715 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" podUID="069b1c9a-1033-46f7-a37b-74a644d6a069" containerName="payload-processing" containerID="cri-o://06f9754fbb6645c9461334551cbdeef750f575312f6b5b5ea0fb9a44f99c0d30" gracePeriod=30 Oct 30 06:20:40.606385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.606244 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:40.657542 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.657503 6999 generic.go:356] "Generic (PLEG): container finished" podID="e010dc1f-4cce-448d-bfea-0e499f8c03a4" containerID="2139307acccc2484cccb6cd24304dc88cdaa7851e59e2c50fbb8d4177cb1a4a2" exitCode=0 Oct 30 06:20:40.657722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.657640 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" event={"ID":"e010dc1f-4cce-448d-bfea-0e499f8c03a4","Type":"ContainerDied","Data":"2139307acccc2484cccb6cd24304dc88cdaa7851e59e2c50fbb8d4177cb1a4a2"} Oct 30 06:20:40.661063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.659654 6999 generic.go:356] "Generic (PLEG): container finished" podID="069b1c9a-1033-46f7-a37b-74a644d6a069" containerID="06f9754fbb6645c9461334551cbdeef750f575312f6b5b5ea0fb9a44f99c0d30" exitCode=0 Oct 30 06:20:40.661063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.659821 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" event={"ID":"069b1c9a-1033-46f7-a37b-74a644d6a069","Type":"ContainerDied","Data":"06f9754fbb6645c9461334551cbdeef750f575312f6b5b5ea0fb9a44f99c0d30"} Oct 30 06:20:40.708268 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.705685 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:20:40.756135 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.756068 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:40.805305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.805271 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e010dc1f-4cce-448d-bfea-0e499f8c03a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e010dc1f-4cce-448d-bfea-0e499f8c03a4-maas-api-tls\") pod \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " Oct 30 06:20:40.805513 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.805485 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e010dc1f-4cce-448d-bfea-0e499f8c03a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e010dc1f-4cce-448d-bfea-0e499f8c03a4-openshift-service-ca\") pod \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " Oct 30 06:20:40.805642 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.805544 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e010dc1f-4cce-448d-bfea-0e499f8c03a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e010dc1f-4cce-448d-bfea-0e499f8c03a4-metrics-certs\") pod \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " Oct 30 06:20:40.805642 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.805623 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e010dc1f-4cce-448d-bfea-0e499f8c03a4-kube-api-access-5c4bm\" (UniqueName: \"kubernetes.io/projected/e010dc1f-4cce-448d-bfea-0e499f8c03a4-kube-api-access-5c4bm\") pod \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\" (UID: \"e010dc1f-4cce-448d-bfea-0e499f8c03a4\") " Oct 30 06:20:40.805923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.805891 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e010dc1f-4cce-448d-bfea-0e499f8c03a4-openshift-service-ca" pod "e010dc1f-4cce-448d-bfea-0e499f8c03a4" (UID: "e010dc1f-4cce-448d-bfea-0e499f8c03a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:40.806436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.806412 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e010dc1f-4cce-448d-bfea-0e499f8c03a4-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:40.807731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.807707 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e010dc1f-4cce-448d-bfea-0e499f8c03a4-maas-api-tls" pod "e010dc1f-4cce-448d-bfea-0e499f8c03a4" (UID: "e010dc1f-4cce-448d-bfea-0e499f8c03a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:40.807895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.807869 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e010dc1f-4cce-448d-bfea-0e499f8c03a4-metrics-certs" pod "e010dc1f-4cce-448d-bfea-0e499f8c03a4" (UID: "e010dc1f-4cce-448d-bfea-0e499f8c03a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:40.808305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.808279 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e010dc1f-4cce-448d-bfea-0e499f8c03a4-kube-api-access-5c4bm" pod "e010dc1f-4cce-448d-bfea-0e499f8c03a4" (UID: "e010dc1f-4cce-448d-bfea-0e499f8c03a4"). InnerVolumeSpecName "kube-api-access-5c4bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:40.817351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.817330 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" Oct 30 06:20:40.922405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.922366 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c4bm\" (UniqueName: \"kubernetes.io/projected/e010dc1f-4cce-448d-bfea-0e499f8c03a4-kube-api-access-5c4bm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:40.922405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.922405 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e010dc1f-4cce-448d-bfea-0e499f8c03a4-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:40.922704 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:40.922421 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e010dc1f-4cce-448d-bfea-0e499f8c03a4-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:41.005056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.005021 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:41.023422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.023380 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069b1c9a-1033-46f7-a37b-74a644d6a069-kube-api-access-pr5bp\" (UniqueName: \"kubernetes.io/projected/069b1c9a-1033-46f7-a37b-74a644d6a069-kube-api-access-pr5bp\") pod \"069b1c9a-1033-46f7-a37b-74a644d6a069\" (UID: \"069b1c9a-1033-46f7-a37b-74a644d6a069\") " Oct 30 06:20:41.023623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.023582 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/069b1c9a-1033-46f7-a37b-74a644d6a069-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069b1c9a-1033-46f7-a37b-74a644d6a069-plugins-config-volume\") pod \"069b1c9a-1033-46f7-a37b-74a644d6a069\" (UID: \"069b1c9a-1033-46f7-a37b-74a644d6a069\") " Oct 30 06:20:41.023982 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.023950 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069b1c9a-1033-46f7-a37b-74a644d6a069-plugins-config-volume" pod "069b1c9a-1033-46f7-a37b-74a644d6a069" (UID: "069b1c9a-1033-46f7-a37b-74a644d6a069"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:41.024177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.024157 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/069b1c9a-1033-46f7-a37b-74a644d6a069-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:41.025787 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.025764 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069b1c9a-1033-46f7-a37b-74a644d6a069-kube-api-access-pr5bp" pod "069b1c9a-1033-46f7-a37b-74a644d6a069" (UID: "069b1c9a-1033-46f7-a37b-74a644d6a069"). InnerVolumeSpecName "kube-api-access-pr5bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:41.125429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.125393 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr5bp\" (UniqueName: \"kubernetes.io/projected/069b1c9a-1033-46f7-a37b-74a644d6a069-kube-api-access-pr5bp\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:41.170373 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.170343 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:41.668636 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.668612 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" Oct 30 06:20:41.669110 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.668643 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv" event={"ID":"069b1c9a-1033-46f7-a37b-74a644d6a069","Type":"ContainerDied","Data":"4f5c65cfda449462add6099b91620afa7b47772be6729b897d96359df88647e8"} Oct 30 06:20:41.669110 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.668715 6999 scope.go:122] "RemoveContainer" containerID="06f9754fbb6645c9461334551cbdeef750f575312f6b5b5ea0fb9a44f99c0d30" Oct 30 06:20:41.670651 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.670627 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" Oct 30 06:20:41.670773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.670708 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97" event={"ID":"e010dc1f-4cce-448d-bfea-0e499f8c03a4","Type":"ContainerDied","Data":"8ef32512990fc70845e2c3621479c16135dcaba73b60f57b24bae01ce875ac2b"} Oct 30 06:20:41.679134 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.679119 6999 scope.go:122] "RemoveContainer" containerID="2139307acccc2484cccb6cd24304dc88cdaa7851e59e2c50fbb8d4177cb1a4a2" Oct 30 06:20:41.721050 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.721020 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97"] Oct 30 06:20:41.732823 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.732794 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f50f6e-6d4b5b6fdc-f4z97"] Oct 30 06:20:41.758335 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.758304 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv"] Oct 30 06:20:41.763094 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.763073 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f50f6e-64658675d-lxsrv"] Oct 30 06:20:41.776437 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:41.776413 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:43.191036 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:43.191000 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:43.401284 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:43.401245 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:43.553876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:43.553803 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:43.609328 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:43.609289 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="069b1c9a-1033-46f7-a37b-74a644d6a069" path="/var/lib/kubelet/pods/069b1c9a-1033-46f7-a37b-74a644d6a069/volumes" Oct 30 06:20:43.609959 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:43.609937 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e010dc1f-4cce-448d-bfea-0e499f8c03a4" path="/var/lib/kubelet/pods/e010dc1f-4cce-448d-bfea-0e499f8c03a4/volumes" Oct 30 06:20:44.865010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:44.864975 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz"] Oct 30 06:20:44.865505 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:44.865164 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" podUID="c7de5845-fa83-4548-9479-dbacf746d212" containerName="maas-api" containerID="cri-o://a0574456e9dc9057330b9f4032553873252e3d86fe839905352f224056d28b14" gracePeriod=30 Oct 30 06:20:44.994632 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:44.994605 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:45.150185 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.150102 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:45.182897 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.182870 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:20:45.278829 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.278793 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7de5845-fa83-4548-9479-dbacf746d212-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7de5845-fa83-4548-9479-dbacf746d212-metrics-certs\") pod \"c7de5845-fa83-4548-9479-dbacf746d212\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " Oct 30 06:20:45.278996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.278840 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7de5845-fa83-4548-9479-dbacf746d212-kube-api-access-dpxtg\" (UniqueName: \"kubernetes.io/projected/c7de5845-fa83-4548-9479-dbacf746d212-kube-api-access-dpxtg\") pod \"c7de5845-fa83-4548-9479-dbacf746d212\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " Oct 30 06:20:45.279056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.279029 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7de5845-fa83-4548-9479-dbacf746d212-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7de5845-fa83-4548-9479-dbacf746d212-openshift-service-ca\") pod \"c7de5845-fa83-4548-9479-dbacf746d212\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " Oct 30 06:20:45.279138 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.279062 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7de5845-fa83-4548-9479-dbacf746d212-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7de5845-fa83-4548-9479-dbacf746d212-maas-api-tls\") pod \"c7de5845-fa83-4548-9479-dbacf746d212\" (UID: \"c7de5845-fa83-4548-9479-dbacf746d212\") " Oct 30 06:20:45.279443 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.279410 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7de5845-fa83-4548-9479-dbacf746d212-openshift-service-ca" pod "c7de5845-fa83-4548-9479-dbacf746d212" (UID: "c7de5845-fa83-4548-9479-dbacf746d212"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:45.279653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.279609 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7de5845-fa83-4548-9479-dbacf746d212-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:45.281172 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.281152 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7de5845-fa83-4548-9479-dbacf746d212-kube-api-access-dpxtg" pod "c7de5845-fa83-4548-9479-dbacf746d212" (UID: "c7de5845-fa83-4548-9479-dbacf746d212"). InnerVolumeSpecName "kube-api-access-dpxtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:45.281458 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.281442 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7de5845-fa83-4548-9479-dbacf746d212-metrics-certs" pod "c7de5845-fa83-4548-9479-dbacf746d212" (UID: "c7de5845-fa83-4548-9479-dbacf746d212"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:45.281546 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.281537 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7de5845-fa83-4548-9479-dbacf746d212-maas-api-tls" pod "c7de5845-fa83-4548-9479-dbacf746d212" (UID: "c7de5845-fa83-4548-9479-dbacf746d212"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:45.296193 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.296165 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:45.384230 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.384182 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7de5845-fa83-4548-9479-dbacf746d212-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:45.384230 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.384227 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7de5845-fa83-4548-9479-dbacf746d212-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:45.394457 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.384245 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpxtg\" (UniqueName: \"kubernetes.io/projected/c7de5845-fa83-4548-9479-dbacf746d212-kube-api-access-dpxtg\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:20:45.656376 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.656347 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:45.698244 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.698191 6999 generic.go:356] "Generic (PLEG): container finished" podID="c7de5845-fa83-4548-9479-dbacf746d212" containerID="a0574456e9dc9057330b9f4032553873252e3d86fe839905352f224056d28b14" exitCode=0 Oct 30 06:20:45.698376 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.698260 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" event={"ID":"c7de5845-fa83-4548-9479-dbacf746d212","Type":"ContainerDied","Data":"a0574456e9dc9057330b9f4032553873252e3d86fe839905352f224056d28b14"} Oct 30 06:20:45.698376 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.698277 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" Oct 30 06:20:45.698376 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.698320 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz" event={"ID":"c7de5845-fa83-4548-9479-dbacf746d212","Type":"ContainerDied","Data":"e11dbe76ce83a182fbf86fa3a7d574745146cb12f0fe228bdde80be178c86311"} Oct 30 06:20:45.698376 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.698351 6999 scope.go:122] "RemoveContainer" containerID="a0574456e9dc9057330b9f4032553873252e3d86fe839905352f224056d28b14" Oct 30 06:20:45.708162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.708131 6999 scope.go:122] "RemoveContainer" containerID="a0574456e9dc9057330b9f4032553873252e3d86fe839905352f224056d28b14" Oct 30 06:20:45.708486 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:20:45.708467 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0574456e9dc9057330b9f4032553873252e3d86fe839905352f224056d28b14\": container with ID starting with a0574456e9dc9057330b9f4032553873252e3d86fe839905352f224056d28b14 not found: ID does not exist" containerID="a0574456e9dc9057330b9f4032553873252e3d86fe839905352f224056d28b14" Oct 30 06:20:45.708558 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.708494 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0574456e9dc9057330b9f4032553873252e3d86fe839905352f224056d28b14"} err="failed to get container status \"a0574456e9dc9057330b9f4032553873252e3d86fe839905352f224056d28b14\": rpc error: code = NotFound desc = could not find container \"a0574456e9dc9057330b9f4032553873252e3d86fe839905352f224056d28b14\": container with ID starting with a0574456e9dc9057330b9f4032553873252e3d86fe839905352f224056d28b14 not found: ID does not exist" Oct 30 06:20:45.762326 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.762304 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz"] Oct 30 06:20:45.773909 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.773884 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-31407e-6d65578dd7-7mgmz"] Oct 30 06:20:45.834535 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.834502 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:45.973042 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:45.972944 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:46.384322 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:46.384290 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:47.609441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:47.609406 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7de5845-fa83-4548-9479-dbacf746d212" path="/var/lib/kubelet/pods/c7de5845-fa83-4548-9479-dbacf746d212/volumes" Oct 30 06:20:48.481552 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:48.481518 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:48.666988 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:48.666959 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:48.868654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:48.868629 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:50.511259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:50.511224 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:51.893125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:51.893093 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:52.076207 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.076150 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:52.280622 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.280525 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj"] Oct 30 06:20:52.281959 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.281902 6999 cpu_manager.go:414] "containerMap: removing container" podUID="179d1033-3d70-4cca-a7cc-0a6bc3223623" containerName="maas-api" Oct 30 06:20:52.282202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.282186 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="179d1033-3d70-4cca-a7cc-0a6bc3223623" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.317922 6999 cpu_manager.go:414] "containerMap: removing container" podUID="e010dc1f-4cce-448d-bfea-0e499f8c03a4" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.317948 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e010dc1f-4cce-448d-bfea-0e499f8c03a4" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.317974 6999 cpu_manager.go:414] "containerMap: removing container" podUID="c7de5845-fa83-4548-9479-dbacf746d212" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.317983 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7de5845-fa83-4548-9479-dbacf746d212" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318009 6999 cpu_manager.go:414] "containerMap: removing container" podUID="47351595-ca08-49ed-bd29-d8046678774a" containerName="payload-pre-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318017 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47351595-ca08-49ed-bd29-d8046678774a" containerName="payload-pre-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318086 6999 cpu_manager.go:414] "containerMap: removing container" podUID="75d39900-c6a7-48e9-8437-725d2bc7f918" containerName="payload-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318094 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d39900-c6a7-48e9-8437-725d2bc7f918" containerName="payload-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318106 6999 cpu_manager.go:414] "containerMap: removing container" podUID="c62893d3-85ac-42d9-a5f4-e7eba0aa5924" containerName="payload-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318114 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c62893d3-85ac-42d9-a5f4-e7eba0aa5924" containerName="payload-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318134 6999 cpu_manager.go:414] "containerMap: removing container" podUID="928adf71-31d6-41fa-88eb-159cc0545a8c" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318141 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="928adf71-31d6-41fa-88eb-159cc0545a8c" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318163 6999 cpu_manager.go:414] "containerMap: removing container" podUID="ace3fa7d-d193-4184-8850-c4741edd37b3" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318170 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace3fa7d-d193-4184-8850-c4741edd37b3" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318187 6999 cpu_manager.go:414] "containerMap: removing container" podUID="069b1c9a-1033-46f7-a37b-74a644d6a069" containerName="payload-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318195 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069b1c9a-1033-46f7-a37b-74a644d6a069" containerName="payload-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318278 6999 cpu_manager.go:414] "containerMap: removing container" podUID="255d0ee6-d1a8-4ccc-a9b9-79343540f145" containerName="istio-proxy" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318286 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255d0ee6-d1a8-4ccc-a9b9-79343540f145" containerName="istio-proxy" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318307 6999 cpu_manager.go:414] "containerMap: removing container" podUID="dff75821-8a85-4c01-a62f-f000b1c4d98b" containerName="istio-proxy" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318314 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff75821-8a85-4c01-a62f-f000b1c4d98b" containerName="istio-proxy" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318336 6999 cpu_manager.go:414] "containerMap: removing container" podUID="8f277c85-c9a4-47ed-9c18-96fcdf78728e" containerName="payload-pre-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318343 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f277c85-c9a4-47ed-9c18-96fcdf78728e" containerName="payload-pre-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318365 6999 cpu_manager.go:414] "containerMap: removing container" podUID="0576b919-84c1-494c-b84a-021af83ccde7" containerName="istio-proxy" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318371 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0576b919-84c1-494c-b84a-021af83ccde7" containerName="istio-proxy" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318401 6999 cpu_manager.go:414] "containerMap: removing container" podUID="3786e3f7-54a2-44b9-a7b0-49c5e333f0f0" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318409 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3786e3f7-54a2-44b9-a7b0-49c5e333f0f0" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318430 6999 cpu_manager.go:414] "containerMap: removing container" podUID="93c24e78-9db8-4c16-b29a-874b93e894dd" containerName="payload-pre-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318436 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93c24e78-9db8-4c16-b29a-874b93e894dd" containerName="payload-pre-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318877 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="179d1033-3d70-4cca-a7cc-0a6bc3223623" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318923 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="e010dc1f-4cce-448d-bfea-0e499f8c03a4" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318939 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7de5845-fa83-4548-9479-dbacf746d212" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318948 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="47351595-ca08-49ed-bd29-d8046678774a" containerName="payload-pre-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318964 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d39900-c6a7-48e9-8437-725d2bc7f918" containerName="payload-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318986 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="c62893d3-85ac-42d9-a5f4-e7eba0aa5924" containerName="payload-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.318997 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="928adf71-31d6-41fa-88eb-159cc0545a8c" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.319013 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace3fa7d-d193-4184-8850-c4741edd37b3" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.319031 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="069b1c9a-1033-46f7-a37b-74a644d6a069" containerName="payload-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.319047 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="255d0ee6-d1a8-4ccc-a9b9-79343540f145" containerName="istio-proxy" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.319065 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff75821-8a85-4c01-a62f-f000b1c4d98b" containerName="istio-proxy" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.319083 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f277c85-c9a4-47ed-9c18-96fcdf78728e" containerName="payload-pre-processing" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.319101 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="0576b919-84c1-494c-b84a-021af83ccde7" containerName="istio-proxy" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.319117 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="3786e3f7-54a2-44b9-a7b0-49c5e333f0f0" containerName="maas-api" Oct 30 06:20:52.319248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.319134 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="93c24e78-9db8-4c16-b29a-874b93e894dd" containerName="payload-pre-processing" Oct 30 06:20:52.359505 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.359482 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj"] Oct 30 06:20:52.359698 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.359682 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" Oct 30 06:20:52.374745 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.374720 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f2cf8a\"" Oct 30 06:20:52.386917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.386894 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f2cf8a-dockercfg-dgdhm\"" Oct 30 06:20:52.424354 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.424324 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9"] Oct 30 06:20:52.428681 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.428662 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" Oct 30 06:20:52.451689 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.451663 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9"] Oct 30 06:20:52.473205 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.473171 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z5rc\" (UniqueName: \"kubernetes.io/projected/744d5495-1eae-487c-92f2-172f6fb29bcf-kube-api-access-5z5rc\") pod \"payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj\" (UID: \"744d5495-1eae-487c-92f2-172f6fb29bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" Oct 30 06:20:52.473381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.473226 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8746x\" (UniqueName: \"kubernetes.io/projected/48f8969a-d9f7-49c0-a00c-f3cd1fe78e08-kube-api-access-8746x\") pod \"payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9\" (UID: \"48f8969a-d9f7-49c0-a00c-f3cd1fe78e08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" Oct 30 06:20:52.473497 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.473474 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744d5495-1eae-487c-92f2-172f6fb29bcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj\" (UID: \"744d5495-1eae-487c-92f2-172f6fb29bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" Oct 30 06:20:52.473683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.473606 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f8969a-d9f7-49c0-a00c-f3cd1fe78e08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9\" (UID: \"48f8969a-d9f7-49c0-a00c-f3cd1fe78e08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" Oct 30 06:20:52.574485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.574402 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744d5495-1eae-487c-92f2-172f6fb29bcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj\" (UID: \"744d5495-1eae-487c-92f2-172f6fb29bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" Oct 30 06:20:52.574485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.574478 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f8969a-d9f7-49c0-a00c-f3cd1fe78e08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9\" (UID: \"48f8969a-d9f7-49c0-a00c-f3cd1fe78e08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" Oct 30 06:20:52.574670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.574562 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z5rc\" (UniqueName: \"kubernetes.io/projected/744d5495-1eae-487c-92f2-172f6fb29bcf-kube-api-access-5z5rc\") pod \"payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj\" (UID: \"744d5495-1eae-487c-92f2-172f6fb29bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" Oct 30 06:20:52.574670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.574588 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8746x\" (UniqueName: \"kubernetes.io/projected/48f8969a-d9f7-49c0-a00c-f3cd1fe78e08-kube-api-access-8746x\") pod \"payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9\" (UID: \"48f8969a-d9f7-49c0-a00c-f3cd1fe78e08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" Oct 30 06:20:52.575164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.575144 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f8969a-d9f7-49c0-a00c-f3cd1fe78e08-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9\" (UID: \"48f8969a-d9f7-49c0-a00c-f3cd1fe78e08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" Oct 30 06:20:52.575236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.575154 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744d5495-1eae-487c-92f2-172f6fb29bcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj\" (UID: \"744d5495-1eae-487c-92f2-172f6fb29bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" Oct 30 06:20:52.598421 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.598398 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z5rc\" (UniqueName: \"kubernetes.io/projected/744d5495-1eae-487c-92f2-172f6fb29bcf-kube-api-access-5z5rc\") pod \"payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj\" (UID: \"744d5495-1eae-487c-92f2-172f6fb29bcf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" Oct 30 06:20:52.598562 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.598476 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8746x\" (UniqueName: \"kubernetes.io/projected/48f8969a-d9f7-49c0-a00c-f3cd1fe78e08-kube-api-access-8746x\") pod \"payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9\" (UID: \"48f8969a-d9f7-49c0-a00c-f3cd1fe78e08\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" Oct 30 06:20:52.676961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.676932 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" Oct 30 06:20:52.743734 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.743587 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" Oct 30 06:20:52.988881 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:52.988847 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj"] Oct 30 06:20:52.989352 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:20:52.989203 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744d5495_1eae_487c_92f2_172f6fb29bcf.slice/crio-6b6fad5a925162454383408af7c884e625bedd201bc50c58a7ea922563b98898 WatchSource:0}: Error finding container 6b6fad5a925162454383408af7c884e625bedd201bc50c58a7ea922563b98898: Status 404 returned error can't find the container with id 6b6fad5a925162454383408af7c884e625bedd201bc50c58a7ea922563b98898 Oct 30 06:20:53.078380 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:53.078360 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9"] Oct 30 06:20:53.080886 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:20:53.080862 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f8969a_d9f7_49c0_a00c_f3cd1fe78e08.slice/crio-e01d753d049cff13ecf8bb7f36bb6499ef6ff8e2d88aae23aeb94c07b6bfe865 WatchSource:0}: Error finding container e01d753d049cff13ecf8bb7f36bb6499ef6ff8e2d88aae23aeb94c07b6bfe865: Status 404 returned error can't find the container with id e01d753d049cff13ecf8bb7f36bb6499ef6ff8e2d88aae23aeb94c07b6bfe865 Oct 30 06:20:53.245292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:53.242645 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:53.758862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:53.758826 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" event={"ID":"48f8969a-d9f7-49c0-a00c-f3cd1fe78e08","Type":"ContainerStarted","Data":"d1f6bfaf646393481209cb0446e8a78d609c19bc1c9c673595ef603b1fb0d5bb"} Oct 30 06:20:53.759061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:53.758883 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" event={"ID":"48f8969a-d9f7-49c0-a00c-f3cd1fe78e08","Type":"ContainerStarted","Data":"e01d753d049cff13ecf8bb7f36bb6499ef6ff8e2d88aae23aeb94c07b6bfe865"} Oct 30 06:20:53.759061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:53.759008 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" Oct 30 06:20:53.760642 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:53.760618 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" event={"ID":"744d5495-1eae-487c-92f2-172f6fb29bcf","Type":"ContainerStarted","Data":"73e4b30066c40c60b8beb8acc0dccb8b4d753982ec460498a86df7f6ebb0ab53"} Oct 30 06:20:53.760781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:53.760660 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" event={"ID":"744d5495-1eae-487c-92f2-172f6fb29bcf","Type":"ContainerStarted","Data":"6b6fad5a925162454383408af7c884e625bedd201bc50c58a7ea922563b98898"} Oct 30 06:20:53.760781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:53.760689 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" Oct 30 06:20:53.801470 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:53.801420 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" podStartSLOduration=1.801403717 podStartE2EDuration="1.801403717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:53.799411586 +0000 UTC m=+1620.308431475" watchObservedRunningTime="2026-10-30 06:20:53.801403717 +0000 UTC m=+1620.310423605" Oct 30 06:20:53.844619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:53.844567 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" podStartSLOduration=1.844553732 podStartE2EDuration="1.844553732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:53.844425368 +0000 UTC m=+1620.353445256" watchObservedRunningTime="2026-10-30 06:20:53.844553732 +0000 UTC m=+1620.353573615" Oct 30 06:20:54.049821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:54.049729 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:54.203412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:54.203375 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:54.487454 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:54.487421 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:55.075797 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:55.075703 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:55.448107 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:55.448068 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:55.756978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:55.756889 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:56.639764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:56.639737 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:57.138707 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:57.138672 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:57.284791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:57.284760 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:57.444898 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:57.444790 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:20:59.675532 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:20:59.675504 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:00.209019 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:00.208989 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:01.429521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:01.429486 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:01.584877 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:01.584845 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:02.202641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:02.202606 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:02.354178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:02.354141 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:03.143809 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:03.143776 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:03.302413 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:03.302377 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:04.142148 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:21:04.142109 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:21:04.238119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:04.238082 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:04.769028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:04.769001 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" Oct 30 06:21:04.769191 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:04.769062 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" Oct 30 06:21:05.863387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:05.863352 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk"] Oct 30 06:21:05.869001 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:05.868979 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:05.888403 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:05.888372 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-520bc4-serving-cert\"" Oct 30 06:21:05.900066 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:05.900040 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk"] Oct 30 06:21:05.980647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:05.980616 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:06.063830 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.063788 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:06.064028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.063848 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn5fk\" (UniqueName: \"kubernetes.io/projected/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-kube-api-access-sn5fk\") pod \"maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:06.064028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.063995 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-metrics-certs\") pod \"maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:06.064272 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.064251 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:06.138122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.138037 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:06.166102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.166069 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:06.166286 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.166151 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:06.166286 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.166174 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn5fk\" (UniqueName: \"kubernetes.io/projected/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-kube-api-access-sn5fk\") pod \"maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:06.166286 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.166193 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-metrics-certs\") pod \"maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:06.166953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.166881 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:06.168736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.168716 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-maas-api-tls\") pod \"maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:06.168822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.168773 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-metrics-certs\") pod \"maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:06.189396 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.189367 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn5fk\" (UniqueName: \"kubernetes.io/projected/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-kube-api-access-sn5fk\") pod \"maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:06.281586 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.281551 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:06.483955 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.483873 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:06.790789 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.790753 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk"] Oct 30 06:21:06.792339 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:21:06.792310 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6b89be_7d33_4cc9_89de_ceeff364e8a4.slice/crio-78305e00d06ce913bdb4e2da4a4f1b5393517635cdbeed01da0b8c49fb359ab6 WatchSource:0}: Error finding container 78305e00d06ce913bdb4e2da4a4f1b5393517635cdbeed01da0b8c49fb359ab6: Status 404 returned error can't find the container with id 78305e00d06ce913bdb4e2da4a4f1b5393517635cdbeed01da0b8c49fb359ab6 Oct 30 06:21:06.843270 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:06.843237 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" event={"ID":"dc6b89be-7d33-4cc9-89de-ceeff364e8a4","Type":"ContainerStarted","Data":"78305e00d06ce913bdb4e2da4a4f1b5393517635cdbeed01da0b8c49fb359ab6"} Oct 30 06:21:07.842361 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:07.842317 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:07.859183 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:07.859144 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" event={"ID":"dc6b89be-7d33-4cc9-89de-ceeff364e8a4","Type":"ContainerStarted","Data":"e6646c25f15446325691c4d25c505de17bf86518c1106b010d1ac09d02e8693b"} Oct 30 06:21:07.859183 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:07.859300 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:07.903002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:07.902954 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" podStartSLOduration=2.557168795 podStartE2EDuration="2.902940325s" totalImagesPullingTime="345.77153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:07.901624975 +0000 UTC m=+1634.410644862" watchObservedRunningTime="2026-10-30 06:21:07.902940325 +0000 UTC m=+1634.411960254" Oct 30 06:21:09.420384 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:09.420351 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9_48f8969a-d9f7-49c0-a00c-f3cd1fe78e08/payload-processing/0.log" Oct 30 06:21:09.507740 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:09.507703 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:09.661966 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:09.661938 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:09.807748 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:09.807655 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:10.061372 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:10.061278 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:10.208382 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:10.208350 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:10.780648 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:10.780620 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:11.474184 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.473995 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn"] Oct 30 06:21:11.478193 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.478169 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.507951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.507927 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-532f3b\"/\"openshift-service-ca.crt\"" Oct 30 06:21:11.507951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.507939 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-532f3b\"/\"kube-root-ca.crt\"" Oct 30 06:21:11.508164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.507979 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-532f3b\"/\"default-dockercfg-7p7n6\"" Oct 30 06:21:11.508164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.508074 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-532f3b\"/\"ipp-route-532f3b-kserve-self-signed-certs\"" Oct 30 06:21:11.511968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.511947 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn"] Oct 30 06:21:11.537913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.537880 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-model-cache\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.538350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.538285 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-tmp-dir\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.538627 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.538606 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-home\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.538848 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.538830 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/165ddac3-4c9c-4712-a5f9-2575ccddae94-tls-certs\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.539035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.539010 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt5jn\" (UniqueName: \"kubernetes.io/projected/165ddac3-4c9c-4712-a5f9-2575ccddae94-kube-api-access-jt5jn\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.539177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.539063 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-dshm\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.567479 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.567449 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:11.640330 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.640291 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-home\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.640504 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.640424 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/165ddac3-4c9c-4712-a5f9-2575ccddae94-tls-certs\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.640569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.640509 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt5jn\" (UniqueName: \"kubernetes.io/projected/165ddac3-4c9c-4712-a5f9-2575ccddae94-kube-api-access-jt5jn\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.640569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.640549 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-dshm\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.640709 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.640687 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-model-cache\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.640784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.640732 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-home\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.640866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.640847 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-tmp-dir\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.641269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.641243 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-tmp-dir\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.641534 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.641511 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-model-cache\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.643522 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.643495 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-dshm\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.643928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.643908 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/165ddac3-4c9c-4712-a5f9-2575ccddae94-tls-certs\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.666573 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.666543 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt5jn\" (UniqueName: \"kubernetes.io/projected/165ddac3-4c9c-4712-a5f9-2575ccddae94-kube-api-access-jt5jn\") pod \"ipp-route-532f3b-kserve-56cf479545-lwxsn\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:11.792658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:11.792556 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:12.124861 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:12.124837 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn"] Oct 30 06:21:12.126835 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:21:12.126806 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod165ddac3_4c9c_4712_a5f9_2575ccddae94.slice/crio-52348e135d66ad1952087f4afadf7bd31f6b9381c917f1940fd7d509a9640084 WatchSource:0}: Error finding container 52348e135d66ad1952087f4afadf7bd31f6b9381c917f1940fd7d509a9640084: Status 404 returned error can't find the container with id 52348e135d66ad1952087f4afadf7bd31f6b9381c917f1940fd7d509a9640084 Oct 30 06:21:12.360269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:12.360204 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:12.928238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:12.928164 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" event={"ID":"165ddac3-4c9c-4712-a5f9-2575ccddae94","Type":"ContainerStarted","Data":"6abd5ff97dd3d199b326548a00c037d2f07c58cac7c5f096dbfb98e8643d2498"} Oct 30 06:21:12.928238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:12.928228 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" event={"ID":"165ddac3-4c9c-4712-a5f9-2575ccddae94","Type":"ContainerStarted","Data":"52348e135d66ad1952087f4afadf7bd31f6b9381c917f1940fd7d509a9640084"} Oct 30 06:21:12.984986 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:12.984933 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" podStartSLOduration=1.984918953 podStartE2EDuration="1.984918953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:12.984000108 +0000 UTC m=+1639.493019995" watchObservedRunningTime="2026-10-30 06:21:12.984918953 +0000 UTC m=+1639.493938840" Oct 30 06:21:13.799243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:13.793088 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:13.871597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:13.871574 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:21:14.749742 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:14.749707 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:16.490907 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:16.490873 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:17.703501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:17.703458 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:17.860587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:17.860545 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:18.142073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:18.142035 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:18.408275 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:18.408178 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:19.901924 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:19.901888 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:20.404808 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.404781 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:20.544825 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.544764 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn"] Oct 30 06:21:20.552552 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.552517 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" podUID="7422fc62-629d-4465-8793-a0821ae7d918" containerName="maas-api" containerID="cri-o://8df5d55ecd6f4d0b99224263fc70b26d815f61be38d3cfab65a1989db36f2b78" gracePeriod=30 Oct 30 06:21:20.641125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.641091 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:20.796173 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.796143 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:20.896611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.896584 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:21:20.943521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.943437 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:20.983885 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.983854 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7422fc62-629d-4465-8793-a0821ae7d918-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7422fc62-629d-4465-8793-a0821ae7d918-maas-api-tls\") pod \"7422fc62-629d-4465-8793-a0821ae7d918\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " Oct 30 06:21:20.984087 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.983962 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7422fc62-629d-4465-8793-a0821ae7d918-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7422fc62-629d-4465-8793-a0821ae7d918-metrics-certs\") pod \"7422fc62-629d-4465-8793-a0821ae7d918\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " Oct 30 06:21:20.984155 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.984115 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7422fc62-629d-4465-8793-a0821ae7d918-kube-api-access-vkhpc\" (UniqueName: \"kubernetes.io/projected/7422fc62-629d-4465-8793-a0821ae7d918-kube-api-access-vkhpc\") pod \"7422fc62-629d-4465-8793-a0821ae7d918\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " Oct 30 06:21:20.984269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.984255 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7422fc62-629d-4465-8793-a0821ae7d918-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7422fc62-629d-4465-8793-a0821ae7d918-openshift-service-ca\") pod \"7422fc62-629d-4465-8793-a0821ae7d918\" (UID: \"7422fc62-629d-4465-8793-a0821ae7d918\") " Oct 30 06:21:20.984879 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.984849 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7422fc62-629d-4465-8793-a0821ae7d918-openshift-service-ca" pod "7422fc62-629d-4465-8793-a0821ae7d918" (UID: "7422fc62-629d-4465-8793-a0821ae7d918"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:20.985003 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.984912 6999 generic.go:356] "Generic (PLEG): container finished" podID="7422fc62-629d-4465-8793-a0821ae7d918" containerID="8df5d55ecd6f4d0b99224263fc70b26d815f61be38d3cfab65a1989db36f2b78" exitCode=0 Oct 30 06:21:20.985003 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.984982 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7422fc62-629d-4465-8793-a0821ae7d918-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:20.985111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.985011 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" Oct 30 06:21:20.985111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.985080 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" event={"ID":"7422fc62-629d-4465-8793-a0821ae7d918","Type":"ContainerDied","Data":"8df5d55ecd6f4d0b99224263fc70b26d815f61be38d3cfab65a1989db36f2b78"} Oct 30 06:21:20.985240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.985154 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn" event={"ID":"7422fc62-629d-4465-8793-a0821ae7d918","Type":"ContainerDied","Data":"1d0f51a97a61b9926a07dc36a3871068c2b4fd175a6bb6ab578fbf9d890fdbfa"} Oct 30 06:21:20.985240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.985187 6999 scope.go:122] "RemoveContainer" containerID="8df5d55ecd6f4d0b99224263fc70b26d815f61be38d3cfab65a1989db36f2b78" Oct 30 06:21:20.987437 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.987401 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7422fc62-629d-4465-8793-a0821ae7d918-maas-api-tls" pod "7422fc62-629d-4465-8793-a0821ae7d918" (UID: "7422fc62-629d-4465-8793-a0821ae7d918"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:20.989389 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.989368 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7422fc62-629d-4465-8793-a0821ae7d918-kube-api-access-vkhpc" pod "7422fc62-629d-4465-8793-a0821ae7d918" (UID: "7422fc62-629d-4465-8793-a0821ae7d918"). InnerVolumeSpecName "kube-api-access-vkhpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:20.989545 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:20.989526 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7422fc62-629d-4465-8793-a0821ae7d918-metrics-certs" pod "7422fc62-629d-4465-8793-a0821ae7d918" (UID: "7422fc62-629d-4465-8793-a0821ae7d918"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:21.019425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.019398 6999 scope.go:122] "RemoveContainer" containerID="8df5d55ecd6f4d0b99224263fc70b26d815f61be38d3cfab65a1989db36f2b78" Oct 30 06:21:21.019808 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:21:21.019785 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df5d55ecd6f4d0b99224263fc70b26d815f61be38d3cfab65a1989db36f2b78\": container with ID starting with 8df5d55ecd6f4d0b99224263fc70b26d815f61be38d3cfab65a1989db36f2b78 not found: ID does not exist" containerID="8df5d55ecd6f4d0b99224263fc70b26d815f61be38d3cfab65a1989db36f2b78" Oct 30 06:21:21.019893 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.019818 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df5d55ecd6f4d0b99224263fc70b26d815f61be38d3cfab65a1989db36f2b78"} err="failed to get container status \"8df5d55ecd6f4d0b99224263fc70b26d815f61be38d3cfab65a1989db36f2b78\": rpc error: code = NotFound desc = could not find container \"8df5d55ecd6f4d0b99224263fc70b26d815f61be38d3cfab65a1989db36f2b78\": container with ID starting with 8df5d55ecd6f4d0b99224263fc70b26d815f61be38d3cfab65a1989db36f2b78 not found: ID does not exist" Oct 30 06:21:21.086666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.086586 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7422fc62-629d-4465-8793-a0821ae7d918-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:21.086666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.086624 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7422fc62-629d-4465-8793-a0821ae7d918-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:21.086666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.086638 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkhpc\" (UniqueName: \"kubernetes.io/projected/7422fc62-629d-4465-8793-a0821ae7d918-kube-api-access-vkhpc\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:21.272647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.272555 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s"] Oct 30 06:21:21.273157 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.273142 6999 cpu_manager.go:414] "containerMap: removing container" podUID="7422fc62-629d-4465-8793-a0821ae7d918" containerName="maas-api" Oct 30 06:21:21.273157 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.273158 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7422fc62-629d-4465-8793-a0821ae7d918" containerName="maas-api" Oct 30 06:21:21.273326 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.273285 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="7422fc62-629d-4465-8793-a0821ae7d918" containerName="maas-api" Oct 30 06:21:21.278281 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.278260 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:21.295485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.295367 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-bbd0e5-serving-cert\"" Oct 30 06:21:21.298141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.298117 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:21.298994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.298975 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s"] Oct 30 06:21:21.363691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.363653 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn"] Oct 30 06:21:21.369883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.369854 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bc5d8f-7f4cfb547d-wpkkn"] Oct 30 06:21:21.391659 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.391627 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0dd6233-fe41-4694-b950-2db40572657d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:21.391853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.391666 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0dd6233-fe41-4694-b950-2db40572657d-metrics-certs\") pod \"maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:21.391853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.391820 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0dd6233-fe41-4694-b950-2db40572657d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:21.391965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.391940 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtlzh\" (UniqueName: \"kubernetes.io/projected/f0dd6233-fe41-4694-b950-2db40572657d-kube-api-access-gtlzh\") pod \"maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:21.481317 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.481271 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:21.493795 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.493767 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0dd6233-fe41-4694-b950-2db40572657d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:21.493992 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.493804 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0dd6233-fe41-4694-b950-2db40572657d-metrics-certs\") pod \"maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:21.493992 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.493888 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0dd6233-fe41-4694-b950-2db40572657d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:21.493992 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.493961 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtlzh\" (UniqueName: \"kubernetes.io/projected/f0dd6233-fe41-4694-b950-2db40572657d-kube-api-access-gtlzh\") pod \"maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:21.494597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.494561 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0dd6233-fe41-4694-b950-2db40572657d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:21.496721 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.496691 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0dd6233-fe41-4694-b950-2db40572657d-metrics-certs\") pod \"maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:21.496822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.496698 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0dd6233-fe41-4694-b950-2db40572657d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:21.549463 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.545999 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtlzh\" (UniqueName: \"kubernetes.io/projected/f0dd6233-fe41-4694-b950-2db40572657d-kube-api-access-gtlzh\") pod \"maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:21.593149 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.593118 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:21.612124 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.612096 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7422fc62-629d-4465-8793-a0821ae7d918" path="/var/lib/kubelet/pods/7422fc62-629d-4465-8793-a0821ae7d918/volumes" Oct 30 06:21:21.704871 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.700595 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:21.793343 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.793305 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:21.811725 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.811641 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:21.921668 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.921639 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s"] Oct 30 06:21:21.922911 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:21:21.922877 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0dd6233_fe41_4694_b950_2db40572657d.slice/crio-7e85319c9e199921c4b25ec058811f48b8b642d29f8a05d791ec2109f8cd5f7d WatchSource:0}: Error finding container 7e85319c9e199921c4b25ec058811f48b8b642d29f8a05d791ec2109f8cd5f7d: Status 404 returned error can't find the container with id 7e85319c9e199921c4b25ec058811f48b8b642d29f8a05d791ec2109f8cd5f7d Oct 30 06:21:21.993020 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:21.992982 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" event={"ID":"f0dd6233-fe41-4694-b950-2db40572657d","Type":"ContainerStarted","Data":"7e85319c9e199921c4b25ec058811f48b8b642d29f8a05d791ec2109f8cd5f7d"} Oct 30 06:21:22.010090 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:22.010059 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:21:22.020593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:22.020565 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:23.002502 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:23.002466 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" event={"ID":"f0dd6233-fe41-4694-b950-2db40572657d","Type":"ContainerStarted","Data":"7c64a379b1b7451c9164c39243d982058ee5bfea3308de7d6eb472be7ed5e2a5"} Oct 30 06:21:23.002946 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:23.002833 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:23.072367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:23.072311 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" podStartSLOduration=1.777820746 podStartE2EDuration="2.072297178s" totalImagesPullingTime="294.476432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:23.070093155 +0000 UTC m=+1649.579113042" watchObservedRunningTime="2026-10-30 06:21:23.072297178 +0000 UTC m=+1649.581317065" Oct 30 06:21:23.643775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:23.643722 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:24.968786 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:24.968746 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:25.118035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:25.117988 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:25.655012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:25.651149 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:26.666124 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:26.666094 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:26.810023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:26.809989 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:27.668043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:27.668013 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:29.015015 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:29.014987 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:21:29.642390 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:29.642352 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:29.796908 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:29.796868 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:30.031681 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:30.031587 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:31.741742 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:31.741701 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:31.906710 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:31.906679 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:32.332565 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:32.332536 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:32.488310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:32.485229 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:33.665731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:33.665700 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:34.141948 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:21:34.141913 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:21:34.607835 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:34.607758 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:36.064433 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:36.064406 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:36.258931 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:36.258896 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:36.409671 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:36.409644 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:36.559452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:36.559414 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:36.715375 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:36.715293 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:37.979835 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:37.978026 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:38.131969 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:38.131932 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:38.453383 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:38.453347 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:40.371193 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:40.371157 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:40.648775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:40.648691 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:42.092174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:42.092147 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:42.265968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:42.265931 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:42.842637 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:42.842597 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w"] Oct 30 06:21:42.842880 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:42.842841 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" podUID="9a7c4be1-b8f3-4c44-81f5-4bd91655897f" containerName="istio-proxy" containerID="cri-o://e02d911e83e6d7c35bba97cc068f620c2936a3af8cdc77520e6b235bb1ef4141" gracePeriod=30 Oct 30 06:21:42.855565 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:42.855537 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn"] Oct 30 06:21:42.855749 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:42.855729 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" podUID="80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" containerName="istio-proxy" containerID="cri-o://e13604ae1476fbf3addca593a6f6408c02453e42175f3966a123e4199772b0bd" gracePeriod=30 Oct 30 06:21:42.857809 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:42.857787 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9"] Oct 30 06:21:42.857985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:42.857966 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" podUID="27965d4b-4c9a-4c68-835f-8f005ff9230d" containerName="istio-proxy" containerID="cri-o://7bf4cd34fb2d7a4baf4145f58948725cba3525f832d93b90d532cd266db831e8" gracePeriod=30 Oct 30 06:21:43.332980 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:43.332944 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:45.332974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:45.332916 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:47.184257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.184225 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9"] Oct 30 06:21:47.188347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.188314 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:47.216895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.216860 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-dfa2ae-serving-cert\"" Oct 30 06:21:47.218405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.218379 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9"] Oct 30 06:21:47.289182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.289149 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:47.302490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.302456 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8acf4a-d7d3-4910-84c6-767d624bf65a-metrics-certs\") pod \"maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:47.302662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.302527 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tbbq\" (UniqueName: \"kubernetes.io/projected/9a8acf4a-d7d3-4910-84c6-767d624bf65a-kube-api-access-9tbbq\") pod \"maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:47.302662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.302620 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8acf4a-d7d3-4910-84c6-767d624bf65a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:47.302776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.302691 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a8acf4a-d7d3-4910-84c6-767d624bf65a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:47.404290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.404252 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8acf4a-d7d3-4910-84c6-767d624bf65a-metrics-certs\") pod \"maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:47.404502 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.404336 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tbbq\" (UniqueName: \"kubernetes.io/projected/9a8acf4a-d7d3-4910-84c6-767d624bf65a-kube-api-access-9tbbq\") pod \"maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:47.404502 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.404420 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8acf4a-d7d3-4910-84c6-767d624bf65a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:47.404629 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.404525 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a8acf4a-d7d3-4910-84c6-767d624bf65a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:47.405065 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.405035 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8acf4a-d7d3-4910-84c6-767d624bf65a-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:47.407079 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.407053 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8acf4a-d7d3-4910-84c6-767d624bf65a-metrics-certs\") pod \"maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:47.407237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.407119 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a8acf4a-d7d3-4910-84c6-767d624bf65a-maas-api-tls\") pod \"maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:47.427640 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.427602 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tbbq\" (UniqueName: \"kubernetes.io/projected/9a8acf4a-d7d3-4910-84c6-767d624bf65a-kube-api-access-9tbbq\") pod \"maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:47.459700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.459583 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:47.503775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.503739 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:47.821697 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:47.821670 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9"] Oct 30 06:21:47.823913 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:21:47.823885 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8acf4a_d7d3_4910_84c6_767d624bf65a.slice/crio-e4b45fa2f7705a846e3dcb64f0358cd022bbe4d7d7d81218c3a14836c471cf96 WatchSource:0}: Error finding container e4b45fa2f7705a846e3dcb64f0358cd022bbe4d7d7d81218c3a14836c471cf96: Status 404 returned error can't find the container with id e4b45fa2f7705a846e3dcb64f0358cd022bbe4d7d7d81218c3a14836c471cf96 Oct 30 06:21:48.172628 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.167698 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:21:48.187966 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.187839 6999 generic.go:356] "Generic (PLEG): container finished" podID="27965d4b-4c9a-4c68-835f-8f005ff9230d" containerID="7bf4cd34fb2d7a4baf4145f58948725cba3525f832d93b90d532cd266db831e8" exitCode=0 Oct 30 06:21:48.187966 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.187913 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" event={"ID":"27965d4b-4c9a-4c68-835f-8f005ff9230d","Type":"ContainerDied","Data":"7bf4cd34fb2d7a4baf4145f58948725cba3525f832d93b90d532cd266db831e8"} Oct 30 06:21:48.189682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.189639 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" event={"ID":"9a8acf4a-d7d3-4910-84c6-767d624bf65a","Type":"ContainerStarted","Data":"e4b45fa2f7705a846e3dcb64f0358cd022bbe4d7d7d81218c3a14836c471cf96"} Oct 30 06:21:48.191747 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.191717 6999 generic.go:356] "Generic (PLEG): container finished" podID="80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" containerID="e13604ae1476fbf3addca593a6f6408c02453e42175f3966a123e4199772b0bd" exitCode=0 Oct 30 06:21:48.191854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.191784 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" event={"ID":"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca","Type":"ContainerDied","Data":"e13604ae1476fbf3addca593a6f6408c02453e42175f3966a123e4199772b0bd"} Oct 30 06:21:48.193350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.193326 6999 generic.go:356] "Generic (PLEG): container finished" podID="9a7c4be1-b8f3-4c44-81f5-4bd91655897f" containerID="e02d911e83e6d7c35bba97cc068f620c2936a3af8cdc77520e6b235bb1ef4141" exitCode=0 Oct 30 06:21:48.193437 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.193402 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" Oct 30 06:21:48.193437 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.193415 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" event={"ID":"9a7c4be1-b8f3-4c44-81f5-4bd91655897f","Type":"ContainerDied","Data":"e02d911e83e6d7c35bba97cc068f620c2936a3af8cdc77520e6b235bb1ef4141"} Oct 30 06:21:48.193563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.193458 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w" event={"ID":"9a7c4be1-b8f3-4c44-81f5-4bd91655897f","Type":"ContainerDied","Data":"d46aaf81e147cf71a2e3a964b0f25c92627da1290266ecfda2d65d8a41a068d4"} Oct 30 06:21:48.193563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.193487 6999 scope.go:122] "RemoveContainer" containerID="e02d911e83e6d7c35bba97cc068f620c2936a3af8cdc77520e6b235bb1ef4141" Oct 30 06:21:48.212711 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.212681 6999 scope.go:122] "RemoveContainer" containerID="e02d911e83e6d7c35bba97cc068f620c2936a3af8cdc77520e6b235bb1ef4141" Oct 30 06:21:48.212970 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:21:48.212951 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02d911e83e6d7c35bba97cc068f620c2936a3af8cdc77520e6b235bb1ef4141\": container with ID starting with e02d911e83e6d7c35bba97cc068f620c2936a3af8cdc77520e6b235bb1ef4141 not found: ID does not exist" containerID="e02d911e83e6d7c35bba97cc068f620c2936a3af8cdc77520e6b235bb1ef4141" Oct 30 06:21:48.213057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.212980 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02d911e83e6d7c35bba97cc068f620c2936a3af8cdc77520e6b235bb1ef4141"} err="failed to get container status \"e02d911e83e6d7c35bba97cc068f620c2936a3af8cdc77520e6b235bb1ef4141\": rpc error: code = NotFound desc = could not find container \"e02d911e83e6d7c35bba97cc068f620c2936a3af8cdc77520e6b235bb1ef4141\": container with ID starting with e02d911e83e6d7c35bba97cc068f620c2936a3af8cdc77520e6b235bb1ef4141 not found: ID does not exist" Oct 30 06:21:48.213421 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.213402 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:21:48.290115 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.290086 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:21:48.323134 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323066 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-credential-socket\") pod \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " Oct 30 06:21:48.323134 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323101 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-token\" (UniqueName: \"kubernetes.io/projected/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-token\") pod \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " Oct 30 06:21:48.323378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323182 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-envoy\") pod \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " Oct 30 06:21:48.323378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323252 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-token\" (UniqueName: \"kubernetes.io/projected/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-token\") pod \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " Oct 30 06:21:48.323378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323269 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-envoy\") pod \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " Oct 30 06:21:48.323378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323298 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-workload-certs\") pod \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " Oct 30 06:21:48.323378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323372 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-credential-socket\") pod \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " Oct 30 06:21:48.323665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323411 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-credential-socket" pod "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" (UID: "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:48.323665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323453 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-workload-socket\") pod \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " Oct 30 06:21:48.323665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323484 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-workload-socket\") pod \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " Oct 30 06:21:48.323665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323519 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-kube-api-access-6ztp4\" (UniqueName: \"kubernetes.io/projected/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-kube-api-access-6ztp4\") pod \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " Oct 30 06:21:48.323665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323573 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istiod-ca-cert\") pod \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " Oct 30 06:21:48.323665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323636 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-kube-api-access-knlpk\" (UniqueName: \"kubernetes.io/projected/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-kube-api-access-knlpk\") pod \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " Oct 30 06:21:48.323981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323675 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-data\") pod \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " Oct 30 06:21:48.323981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323717 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-workload-certs\") pod \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " Oct 30 06:21:48.323981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323778 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-workload-socket" pod "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" (UID: "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:48.323981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323793 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-podinfo\") pod \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " Oct 30 06:21:48.323981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323842 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-podinfo\") pod \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\" (UID: \"9a7c4be1-b8f3-4c44-81f5-4bd91655897f\") " Oct 30 06:21:48.323981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323922 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-data\") pod \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " Oct 30 06:21:48.323981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323956 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-workload-certs" pod "9a7c4be1-b8f3-4c44-81f5-4bd91655897f" (UID: "9a7c4be1-b8f3-4c44-81f5-4bd91655897f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:48.323981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.323960 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istiod-ca-cert\") pod \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\" (UID: \"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca\") " Oct 30 06:21:48.325979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.324242 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istiod-ca-cert" pod "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" (UID: "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:48.325979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.324477 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-credential-socket" pod "9a7c4be1-b8f3-4c44-81f5-4bd91655897f" (UID: "9a7c4be1-b8f3-4c44-81f5-4bd91655897f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:48.325979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.324829 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-workload-socket" pod "9a7c4be1-b8f3-4c44-81f5-4bd91655897f" (UID: "9a7c4be1-b8f3-4c44-81f5-4bd91655897f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:48.325979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.324911 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.325979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.324924 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.325979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.324932 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.325979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.324944 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.325979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.324958 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.325979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.324965 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.325979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.325654 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-workload-certs" pod "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" (UID: "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:48.326726 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.326693 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-envoy" pod "9a7c4be1-b8f3-4c44-81f5-4bd91655897f" (UID: "9a7c4be1-b8f3-4c44-81f5-4bd91655897f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:48.326995 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.326979 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-token" pod "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" (UID: "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:48.327573 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.327534 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-envoy" pod "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" (UID: "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:48.327989 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.327953 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-data" pod "9a7c4be1-b8f3-4c44-81f5-4bd91655897f" (UID: "9a7c4be1-b8f3-4c44-81f5-4bd91655897f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:48.328091 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.328020 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-data" pod "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" (UID: "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:48.329260 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.329230 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-token" pod "9a7c4be1-b8f3-4c44-81f5-4bd91655897f" (UID: "9a7c4be1-b8f3-4c44-81f5-4bd91655897f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:48.329426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.329407 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-podinfo" pod "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" (UID: "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:48.329508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.329440 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istiod-ca-cert" pod "9a7c4be1-b8f3-4c44-81f5-4bd91655897f" (UID: "9a7c4be1-b8f3-4c44-81f5-4bd91655897f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:48.329754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.329735 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-kube-api-access-6ztp4" pod "9a7c4be1-b8f3-4c44-81f5-4bd91655897f" (UID: "9a7c4be1-b8f3-4c44-81f5-4bd91655897f"). InnerVolumeSpecName "kube-api-access-6ztp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:48.330107 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.330069 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-kube-api-access-knlpk" pod "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" (UID: "80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca"). InnerVolumeSpecName "kube-api-access-knlpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:48.330832 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.330810 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-podinfo" pod "9a7c4be1-b8f3-4c44-81f5-4bd91655897f" (UID: "9a7c4be1-b8f3-4c44-81f5-4bd91655897f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:48.427114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.426097 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27965d4b-4c9a-4c68-835f-8f005ff9230d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27965d4b-4c9a-4c68-835f-8f005ff9230d-istiod-ca-cert\") pod \"27965d4b-4c9a-4c68-835f-8f005ff9230d\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " Oct 30 06:21:48.427114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.426149 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-token\" (UniqueName: \"kubernetes.io/projected/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-token\") pod \"27965d4b-4c9a-4c68-835f-8f005ff9230d\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " Oct 30 06:21:48.427114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.426278 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-podinfo\") pod \"27965d4b-4c9a-4c68-835f-8f005ff9230d\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " Oct 30 06:21:48.427114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.426326 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-workload-socket\") pod \"27965d4b-4c9a-4c68-835f-8f005ff9230d\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " Oct 30 06:21:48.427114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.426375 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-workload-certs\") pod \"27965d4b-4c9a-4c68-835f-8f005ff9230d\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " Oct 30 06:21:48.427114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.426430 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-credential-socket\") pod \"27965d4b-4c9a-4c68-835f-8f005ff9230d\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " Oct 30 06:21:48.427114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.426517 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-envoy\") pod \"27965d4b-4c9a-4c68-835f-8f005ff9230d\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " Oct 30 06:21:48.427114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.426572 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27965d4b-4c9a-4c68-835f-8f005ff9230d-kube-api-access-78xxc\" (UniqueName: \"kubernetes.io/projected/27965d4b-4c9a-4c68-835f-8f005ff9230d-kube-api-access-78xxc\") pod \"27965d4b-4c9a-4c68-835f-8f005ff9230d\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " Oct 30 06:21:48.427114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.426681 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-data\") pod \"27965d4b-4c9a-4c68-835f-8f005ff9230d\" (UID: \"27965d4b-4c9a-4c68-835f-8f005ff9230d\") " Oct 30 06:21:48.427661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.427138 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-credential-socket" pod "27965d4b-4c9a-4c68-835f-8f005ff9230d" (UID: "27965d4b-4c9a-4c68-835f-8f005ff9230d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:48.427661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.427249 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ztp4\" (UniqueName: \"kubernetes.io/projected/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-kube-api-access-6ztp4\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.427661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.427270 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.427661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.427284 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knlpk\" (UniqueName: \"kubernetes.io/projected/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-kube-api-access-knlpk\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.427661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.427296 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.427661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.427307 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.427661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.427320 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.427661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.427331 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.427661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.427342 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.427661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.427355 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.427661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.427366 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.427661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.427376 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.427661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.427388 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7c4be1-b8f3-4c44-81f5-4bd91655897f-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.427661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.427575 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-data" pod "27965d4b-4c9a-4c68-835f-8f005ff9230d" (UID: "27965d4b-4c9a-4c68-835f-8f005ff9230d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:48.429477 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.429442 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-workload-certs" pod "27965d4b-4c9a-4c68-835f-8f005ff9230d" (UID: "27965d4b-4c9a-4c68-835f-8f005ff9230d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:48.429599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.429567 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27965d4b-4c9a-4c68-835f-8f005ff9230d-istiod-ca-cert" pod "27965d4b-4c9a-4c68-835f-8f005ff9230d" (UID: "27965d4b-4c9a-4c68-835f-8f005ff9230d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:48.429659 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.429611 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-workload-socket" pod "27965d4b-4c9a-4c68-835f-8f005ff9230d" (UID: "27965d4b-4c9a-4c68-835f-8f005ff9230d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:48.430308 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.430267 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-token" pod "27965d4b-4c9a-4c68-835f-8f005ff9230d" (UID: "27965d4b-4c9a-4c68-835f-8f005ff9230d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:48.430568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.430543 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27965d4b-4c9a-4c68-835f-8f005ff9230d-kube-api-access-78xxc" pod "27965d4b-4c9a-4c68-835f-8f005ff9230d" (UID: "27965d4b-4c9a-4c68-835f-8f005ff9230d"). InnerVolumeSpecName "kube-api-access-78xxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:48.430786 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.430752 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-podinfo" pod "27965d4b-4c9a-4c68-835f-8f005ff9230d" (UID: "27965d4b-4c9a-4c68-835f-8f005ff9230d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:48.433369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.433343 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-envoy" pod "27965d4b-4c9a-4c68-835f-8f005ff9230d" (UID: "27965d4b-4c9a-4c68-835f-8f005ff9230d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:48.528727 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.528702 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.528727 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.528726 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.528901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.528736 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.528901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.528744 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.528901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.528753 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.528901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.528760 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78xxc\" (UniqueName: \"kubernetes.io/projected/27965d4b-4c9a-4c68-835f-8f005ff9230d-kube-api-access-78xxc\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.528901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.528768 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.528901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.528776 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27965d4b-4c9a-4c68-835f-8f005ff9230d-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.528901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.528784 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27965d4b-4c9a-4c68-835f-8f005ff9230d-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.549907 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.549867 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w"] Oct 30 06:21:48.554624 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.554598 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-54g5w"] Oct 30 06:21:48.825771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:48.825732 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:49.204625 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.204595 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" Oct 30 06:21:49.205416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.204627 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9" event={"ID":"27965d4b-4c9a-4c68-835f-8f005ff9230d","Type":"ContainerDied","Data":"f89601088e05b90a2c296fa1b40e574f912ee4cb87cd1ce22602c71bfba80a1f"} Oct 30 06:21:49.205554 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.205481 6999 scope.go:122] "RemoveContainer" containerID="7bf4cd34fb2d7a4baf4145f58948725cba3525f832d93b90d532cd266db831e8" Oct 30 06:21:49.207627 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.207601 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" event={"ID":"9a8acf4a-d7d3-4910-84c6-767d624bf65a","Type":"ContainerStarted","Data":"1e5c3f63f7cb97a6b162b41557060563ceb22823aeae853fa10063933f8a5c1b"} Oct 30 06:21:49.207782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.207766 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:49.209994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.209967 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" event={"ID":"80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca","Type":"ContainerDied","Data":"3ea5b1c2995e19c82305f7d0c8e3974eff74eae832223f41a77090d860fd54fd"} Oct 30 06:21:49.210102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.210061 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn" Oct 30 06:21:49.224945 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.224148 6999 scope.go:122] "RemoveContainer" containerID="e13604ae1476fbf3addca593a6f6408c02453e42175f3966a123e4199772b0bd" Oct 30 06:21:49.286887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.286540 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" podStartSLOduration=1.917777549 podStartE2EDuration="2.286520397s" totalImagesPullingTime="368.742848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:49.283373155 +0000 UTC m=+1675.792393045" watchObservedRunningTime="2026-10-30 06:21:49.286520397 +0000 UTC m=+1675.795540287" Oct 30 06:21:49.342988 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.342960 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn"] Oct 30 06:21:49.345980 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.345942 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:49.349116 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.349094 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rrwcn"] Oct 30 06:21:49.385039 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.385008 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9"] Oct 30 06:21:49.390313 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.390291 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-q47h9"] Oct 30 06:21:49.518841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.518095 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:49.609963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.609933 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27965d4b-4c9a-4c68-835f-8f005ff9230d" path="/var/lib/kubelet/pods/27965d4b-4c9a-4c68-835f-8f005ff9230d/volumes" Oct 30 06:21:49.610641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.610621 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" path="/var/lib/kubelet/pods/80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca/volumes" Oct 30 06:21:49.611114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:49.611101 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7c4be1-b8f3-4c44-81f5-4bd91655897f" path="/var/lib/kubelet/pods/9a7c4be1-b8f3-4c44-81f5-4bd91655897f/volumes" Oct 30 06:21:51.443346 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:51.443308 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:53.435611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:53.435571 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:53.631238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:53.630281 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:55.224414 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.224387 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:21:55.411314 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.411282 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:55.598906 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.598824 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z"] Oct 30 06:21:55.599752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.599725 6999 cpu_manager.go:414] "containerMap: removing container" podUID="27965d4b-4c9a-4c68-835f-8f005ff9230d" containerName="istio-proxy" Oct 30 06:21:55.599752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.599754 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27965d4b-4c9a-4c68-835f-8f005ff9230d" containerName="istio-proxy" Oct 30 06:21:55.600035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.599775 6999 cpu_manager.go:414] "containerMap: removing container" podUID="80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" containerName="istio-proxy" Oct 30 06:21:55.600035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.599783 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" containerName="istio-proxy" Oct 30 06:21:55.600035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.599797 6999 cpu_manager.go:414] "containerMap: removing container" podUID="9a7c4be1-b8f3-4c44-81f5-4bd91655897f" containerName="istio-proxy" Oct 30 06:21:55.600035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.599805 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7c4be1-b8f3-4c44-81f5-4bd91655897f" containerName="istio-proxy" Oct 30 06:21:55.600035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.599979 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="27965d4b-4c9a-4c68-835f-8f005ff9230d" containerName="istio-proxy" Oct 30 06:21:55.600035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.599996 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="80eaf9bc-b46e-4bf7-a81e-f6f55b9994ca" containerName="istio-proxy" Oct 30 06:21:55.600035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.600004 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7c4be1-b8f3-4c44-81f5-4bd91655897f" containerName="istio-proxy" Oct 30 06:21:55.606197 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.606174 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" Oct 30 06:21:55.631000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.630971 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z"] Oct 30 06:21:55.631307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.631281 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b95450c7-2660-451e-b701-bd5a43637389-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z\" (UID: \"b95450c7-2660-451e-b701-bd5a43637389\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" Oct 30 06:21:55.631416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.631331 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4jmw\" (UniqueName: \"kubernetes.io/projected/b95450c7-2660-451e-b701-bd5a43637389-kube-api-access-j4jmw\") pod \"payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z\" (UID: \"b95450c7-2660-451e-b701-bd5a43637389\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" Oct 30 06:21:55.637250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.637190 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:55.707238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.698245 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-f6a022\"" Oct 30 06:21:55.707238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.698298 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-f6a022-dockercfg-ls5vh\"" Oct 30 06:21:55.733105 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.733069 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b95450c7-2660-451e-b701-bd5a43637389-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z\" (UID: \"b95450c7-2660-451e-b701-bd5a43637389\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" Oct 30 06:21:55.733341 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.733113 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4jmw\" (UniqueName: \"kubernetes.io/projected/b95450c7-2660-451e-b701-bd5a43637389-kube-api-access-j4jmw\") pod \"payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z\" (UID: \"b95450c7-2660-451e-b701-bd5a43637389\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" Oct 30 06:21:55.733839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.733814 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b95450c7-2660-451e-b701-bd5a43637389-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z\" (UID: \"b95450c7-2660-451e-b701-bd5a43637389\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" Oct 30 06:21:55.756974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.756948 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4jmw\" (UniqueName: \"kubernetes.io/projected/b95450c7-2660-451e-b701-bd5a43637389-kube-api-access-j4jmw\") pod \"payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z\" (UID: \"b95450c7-2660-451e-b701-bd5a43637389\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" Oct 30 06:21:55.794775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.794744 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:55.923194 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:55.923154 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" Oct 30 06:21:56.241053 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:56.241001 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z"] Oct 30 06:21:56.244020 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:21:56.243968 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95450c7_2660_451e_b701_bd5a43637389.slice/crio-f168e51e1ab2f7ea79a6d756611a7e1c05b7bc74f7b68289a29b96c6b175f817 WatchSource:0}: Error finding container f168e51e1ab2f7ea79a6d756611a7e1c05b7bc74f7b68289a29b96c6b175f817: Status 404 returned error can't find the container with id f168e51e1ab2f7ea79a6d756611a7e1c05b7bc74f7b68289a29b96c6b175f817 Oct 30 06:21:56.262710 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:56.262584 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" event={"ID":"b95450c7-2660-451e-b701-bd5a43637389","Type":"ContainerStarted","Data":"f168e51e1ab2f7ea79a6d756611a7e1c05b7bc74f7b68289a29b96c6b175f817"} Oct 30 06:21:57.270578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:57.270536 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" event={"ID":"b95450c7-2660-451e-b701-bd5a43637389","Type":"ContainerStarted","Data":"ade620236888d137b8728b7fd1ea1d6a3194c9d685ad46e9e268dbabefce46d5"} Oct 30 06:21:57.271021 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:57.270605 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" Oct 30 06:21:57.311559 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:57.311502 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" podStartSLOduration=2.311487287 podStartE2EDuration="2.311487287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:57.310647751 +0000 UTC m=+1683.819667650" watchObservedRunningTime="2026-10-30 06:21:57.311487287 +0000 UTC m=+1683.820507173" Oct 30 06:21:57.584917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:57.584818 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:59.770423 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.770390 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6"] Oct 30 06:21:59.774612 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.774590 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:21:59.807521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.807478 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-93c22e-serving-cert\"" Oct 30 06:21:59.807792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.807769 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6"] Oct 30 06:21:59.886838 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.886803 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:21:59.887030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.887007 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-metrics-certs\") pod \"maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:21:59.887083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.887043 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvk7q\" (UniqueName: \"kubernetes.io/projected/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-kube-api-access-cvk7q\") pod \"maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:21:59.887172 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.887152 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:21:59.939876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.939844 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:21:59.987975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.987935 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:21:59.988164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.988003 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:21:59.988164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.988144 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-metrics-certs\") pod \"maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:21:59.988308 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.988173 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvk7q\" (UniqueName: \"kubernetes.io/projected/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-kube-api-access-cvk7q\") pod \"maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:21:59.988708 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.988683 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:21:59.990587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.990567 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-maas-api-tls\") pod \"maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:21:59.990691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:21:59.990630 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-metrics-certs\") pod \"maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:22:00.012115 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:00.012087 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvk7q\" (UniqueName: \"kubernetes.io/projected/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-kube-api-access-cvk7q\") pod \"maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:22:00.088102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:00.087957 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:00.088102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:00.088013 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:22:00.364363 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:00.364335 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:00.408978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:00.408953 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6"] Oct 30 06:22:00.410942 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:22:00.410916 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2806ba44_9515_4f41_a30f_f3aa24fe9aaf.slice/crio-9049380a145190a61921af8bd2226d3ae5eec1b60e268f4c75490d4ce10e228c WatchSource:0}: Error finding container 9049380a145190a61921af8bd2226d3ae5eec1b60e268f4c75490d4ce10e228c: Status 404 returned error can't find the container with id 9049380a145190a61921af8bd2226d3ae5eec1b60e268f4c75490d4ce10e228c Oct 30 06:22:01.297898 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:01.297864 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" event={"ID":"2806ba44-9515-4f41-a30f-f3aa24fe9aaf","Type":"ContainerStarted","Data":"59566472b9fb54877c15ecf912ae2d23e9fe3ef75af79dc40d2eeb09ffec366e"} Oct 30 06:22:01.298423 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:01.297922 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" event={"ID":"2806ba44-9515-4f41-a30f-f3aa24fe9aaf","Type":"ContainerStarted","Data":"9049380a145190a61921af8bd2226d3ae5eec1b60e268f4c75490d4ce10e228c"} Oct 30 06:22:01.298423 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:01.298093 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:22:01.345996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:01.345937 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" podStartSLOduration=1.987738726 podStartE2EDuration="2.345920622s" totalImagesPullingTime="358.181896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:01.343059781 +0000 UTC m=+1687.852079670" watchObservedRunningTime="2026-10-30 06:22:01.345920622 +0000 UTC m=+1687.854940509" Oct 30 06:22:01.769045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:01.769014 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:02.448710 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:02.448682 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:03.755150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:03.755115 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:04.141942 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:22:04.141906 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:22:05.362077 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.362042 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb"] Oct 30 06:22:05.366546 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.366524 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:05.400032 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.399995 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb"] Oct 30 06:22:05.436269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.436242 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-4dd310-serving-cert\"" Oct 30 06:22:05.474975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.474930 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05415885-73d0-40f6-a332-e4dd345f55d2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:05.475156 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.475001 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:05.475156 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.475097 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7hqj\" (UniqueName: \"kubernetes.io/projected/05415885-73d0-40f6-a332-e4dd345f55d2-kube-api-access-b7hqj\") pod \"maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:05.475282 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.475161 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-metrics-certs\") pod \"maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:05.493181 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.493145 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:05.576738 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.576707 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05415885-73d0-40f6-a332-e4dd345f55d2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:05.576963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.576766 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:05.576963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.576811 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7hqj\" (UniqueName: \"kubernetes.io/projected/05415885-73d0-40f6-a332-e4dd345f55d2-kube-api-access-b7hqj\") pod \"maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:05.576963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.576877 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-metrics-certs\") pod \"maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:05.577155 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:22:05.576992 6999 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-serving-cert: secret "maas-api-e2e-praxis-b-4dd310-serving-cert" not found Oct 30 06:22:05.577155 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:22:05.577067 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-maas-api-tls podName:05415885-73d0-40f6-a332-e4dd345f55d2 nodeName:}" failed. No retries permitted until 2026-10-30 06:22:06.077049908 +0000 UTC m=+1692.586069773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-maas-api-tls") pod "maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" (UID: "05415885-73d0-40f6-a332-e4dd345f55d2") : secret "maas-api-e2e-praxis-b-4dd310-serving-cert" not found Oct 30 06:22:05.577554 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.577525 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05415885-73d0-40f6-a332-e4dd345f55d2-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:05.579898 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.579876 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-metrics-certs\") pod \"maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:05.601574 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.601536 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7hqj\" (UniqueName: \"kubernetes.io/projected/05415885-73d0-40f6-a332-e4dd345f55d2-kube-api-access-b7hqj\") pod \"maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:05.643620 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.643536 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:05.789831 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:05.789802 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:06.083317 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:06.083181 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:06.085843 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:06.085819 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-maas-api-tls\") pod \"maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:06.280710 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:06.280675 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:06.623201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:06.620786 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb"] Oct 30 06:22:06.623201 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:22:06.622611 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05415885_73d0_40f6_a332_e4dd345f55d2.slice/crio-328cf0bbdbd938d66ed750dcd3087d958f56d4f011a32b39b7a274cc7adfd42f WatchSource:0}: Error finding container 328cf0bbdbd938d66ed750dcd3087d958f56d4f011a32b39b7a274cc7adfd42f: Status 404 returned error can't find the container with id 328cf0bbdbd938d66ed750dcd3087d958f56d4f011a32b39b7a274cc7adfd42f Oct 30 06:22:06.879967 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:06.879864 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h"] Oct 30 06:22:06.880162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:06.880117 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" podUID="a6aefe60-58f8-490e-9c3d-e062edf6e038" containerName="istio-proxy" containerID="cri-o://4f4e9763b521bf0750a5af508be168427035d62e36b09536a0ca1bc024e38c8a" gracePeriod=30 Oct 30 06:22:07.310499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:07.310422 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:22:07.344799 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:07.344763 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" event={"ID":"05415885-73d0-40f6-a332-e4dd345f55d2","Type":"ContainerStarted","Data":"ecd9b9f0c3603cf4e5bdcca8a4e0628ab3a5ca213f6179538d590919a2ef58d0"} Oct 30 06:22:07.344981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:07.344819 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" event={"ID":"05415885-73d0-40f6-a332-e4dd345f55d2","Type":"ContainerStarted","Data":"328cf0bbdbd938d66ed750dcd3087d958f56d4f011a32b39b7a274cc7adfd42f"} Oct 30 06:22:07.344981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:07.344860 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:07.404439 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:07.404327 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" podStartSLOduration=2.035167727 podStartE2EDuration="2.404307896s" totalImagesPullingTime="369.140169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:07.401533876 +0000 UTC m=+1693.910553764" watchObservedRunningTime="2026-10-30 06:22:07.404307896 +0000 UTC m=+1693.913327780" Oct 30 06:22:07.459448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:07.459413 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:08.280392 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:08.280365 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" Oct 30 06:22:09.821866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:09.821835 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:11.694654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:11.694609 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:22:11.695072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:11.694678 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" podUID="a6aefe60-58f8-490e-9c3d-e062edf6e038" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:22:11.724654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:11.723352 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:12.169940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.169904 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:22:12.276493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.276193 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-credential-socket\") pod \"a6aefe60-58f8-490e-9c3d-e062edf6e038\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " Oct 30 06:22:12.276493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.276309 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-envoy\") pod \"a6aefe60-58f8-490e-9c3d-e062edf6e038\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " Oct 30 06:22:12.276493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.276353 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6aefe60-58f8-490e-9c3d-e062edf6e038-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6aefe60-58f8-490e-9c3d-e062edf6e038-istiod-ca-cert\") pod \"a6aefe60-58f8-490e-9c3d-e062edf6e038\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " Oct 30 06:22:12.276493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.276475 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-token\" (UniqueName: \"kubernetes.io/projected/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-token\") pod \"a6aefe60-58f8-490e-9c3d-e062edf6e038\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " Oct 30 06:22:12.276493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.276496 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-workload-certs\") pod \"a6aefe60-58f8-490e-9c3d-e062edf6e038\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " Oct 30 06:22:12.276915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.276554 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-credential-socket" pod "a6aefe60-58f8-490e-9c3d-e062edf6e038" (UID: "a6aefe60-58f8-490e-9c3d-e062edf6e038"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:12.276915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.276637 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-data\") pod \"a6aefe60-58f8-490e-9c3d-e062edf6e038\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " Oct 30 06:22:12.276915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.276684 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6aefe60-58f8-490e-9c3d-e062edf6e038-kube-api-access-jq8cd\" (UniqueName: \"kubernetes.io/projected/a6aefe60-58f8-490e-9c3d-e062edf6e038-kube-api-access-jq8cd\") pod \"a6aefe60-58f8-490e-9c3d-e062edf6e038\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " Oct 30 06:22:12.276915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.276725 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-workload-socket\") pod \"a6aefe60-58f8-490e-9c3d-e062edf6e038\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " Oct 30 06:22:12.276915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.276797 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-podinfo\") pod \"a6aefe60-58f8-490e-9c3d-e062edf6e038\" (UID: \"a6aefe60-58f8-490e-9c3d-e062edf6e038\") " Oct 30 06:22:12.276915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.276802 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-workload-certs" pod "a6aefe60-58f8-490e-9c3d-e062edf6e038" (UID: "a6aefe60-58f8-490e-9c3d-e062edf6e038"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:12.276915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.276901 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6aefe60-58f8-490e-9c3d-e062edf6e038-istiod-ca-cert" pod "a6aefe60-58f8-490e-9c3d-e062edf6e038" (UID: "a6aefe60-58f8-490e-9c3d-e062edf6e038"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:12.277462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.277154 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-workload-socket" pod "a6aefe60-58f8-490e-9c3d-e062edf6e038" (UID: "a6aefe60-58f8-490e-9c3d-e062edf6e038"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:12.277462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.277400 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-data" pod "a6aefe60-58f8-490e-9c3d-e062edf6e038" (UID: "a6aefe60-58f8-490e-9c3d-e062edf6e038"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:12.277586 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.277536 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:12.277586 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.277554 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:12.277586 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.277568 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:12.277586 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.277583 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6aefe60-58f8-490e-9c3d-e062edf6e038-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:12.277802 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.277596 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:12.279745 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.279701 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-token" pod "a6aefe60-58f8-490e-9c3d-e062edf6e038" (UID: "a6aefe60-58f8-490e-9c3d-e062edf6e038"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:12.280030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.280007 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6aefe60-58f8-490e-9c3d-e062edf6e038-kube-api-access-jq8cd" pod "a6aefe60-58f8-490e-9c3d-e062edf6e038" (UID: "a6aefe60-58f8-490e-9c3d-e062edf6e038"). InnerVolumeSpecName "kube-api-access-jq8cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:12.280525 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.280492 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-envoy" pod "a6aefe60-58f8-490e-9c3d-e062edf6e038" (UID: "a6aefe60-58f8-490e-9c3d-e062edf6e038"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:12.288907 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.288860 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-podinfo" pod "a6aefe60-58f8-490e-9c3d-e062edf6e038" (UID: "a6aefe60-58f8-490e-9c3d-e062edf6e038"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:12.379091 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.379047 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq8cd\" (UniqueName: \"kubernetes.io/projected/a6aefe60-58f8-490e-9c3d-e062edf6e038-kube-api-access-jq8cd\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:12.379091 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.379092 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:12.379364 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.379108 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:12.379364 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.379120 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6aefe60-58f8-490e-9c3d-e062edf6e038-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:12.394791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.394752 6999 generic.go:356] "Generic (PLEG): container finished" podID="a6aefe60-58f8-490e-9c3d-e062edf6e038" containerID="4f4e9763b521bf0750a5af508be168427035d62e36b09536a0ca1bc024e38c8a" exitCode=0 Oct 30 06:22:12.394791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.394799 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" Oct 30 06:22:12.395027 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.394844 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" event={"ID":"a6aefe60-58f8-490e-9c3d-e062edf6e038","Type":"ContainerDied","Data":"4f4e9763b521bf0750a5af508be168427035d62e36b09536a0ca1bc024e38c8a"} Oct 30 06:22:12.395027 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.394908 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h" event={"ID":"a6aefe60-58f8-490e-9c3d-e062edf6e038","Type":"ContainerDied","Data":"b262ddd98600a274b2e7bfc1a5d9914e4a8b5dbbbc1cd4c98545f327cc43dcc0"} Oct 30 06:22:12.395027 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.394937 6999 scope.go:122] "RemoveContainer" containerID="4f4e9763b521bf0750a5af508be168427035d62e36b09536a0ca1bc024e38c8a" Oct 30 06:22:12.407773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.407752 6999 scope.go:122] "RemoveContainer" containerID="4f4e9763b521bf0750a5af508be168427035d62e36b09536a0ca1bc024e38c8a" Oct 30 06:22:12.408208 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:22:12.408187 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f4e9763b521bf0750a5af508be168427035d62e36b09536a0ca1bc024e38c8a\": container with ID starting with 4f4e9763b521bf0750a5af508be168427035d62e36b09536a0ca1bc024e38c8a not found: ID does not exist" containerID="4f4e9763b521bf0750a5af508be168427035d62e36b09536a0ca1bc024e38c8a" Oct 30 06:22:12.408208 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.408235 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f4e9763b521bf0750a5af508be168427035d62e36b09536a0ca1bc024e38c8a"} err="failed to get container status \"4f4e9763b521bf0750a5af508be168427035d62e36b09536a0ca1bc024e38c8a\": rpc error: code = NotFound desc = could not find container \"4f4e9763b521bf0750a5af508be168427035d62e36b09536a0ca1bc024e38c8a\": container with ID starting with 4f4e9763b521bf0750a5af508be168427035d62e36b09536a0ca1bc024e38c8a not found: ID does not exist" Oct 30 06:22:12.449638 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.449592 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h"] Oct 30 06:22:12.453775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:12.453741 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-dls9h"] Oct 30 06:22:13.358621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:13.358594 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:22:13.608994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:13.608911 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6aefe60-58f8-490e-9c3d-e062edf6e038" path="/var/lib/kubelet/pods/a6aefe60-58f8-490e-9c3d-e062edf6e038/volumes" Oct 30 06:22:13.659860 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:13.659822 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:15.562126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:15.562089 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:16.043759 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:16.043727 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:17.373526 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:17.373488 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:17.520874 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:17.520833 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:17.864586 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:17.864543 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:18.013177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:18.013146 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:19.153671 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.153634 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c"] Oct 30 06:22:19.154393 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.154369 6999 cpu_manager.go:414] "containerMap: removing container" podUID="a6aefe60-58f8-490e-9c3d-e062edf6e038" containerName="istio-proxy" Oct 30 06:22:19.154393 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.154392 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6aefe60-58f8-490e-9c3d-e062edf6e038" containerName="istio-proxy" Oct 30 06:22:19.154589 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.154567 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6aefe60-58f8-490e-9c3d-e062edf6e038" containerName="istio-proxy" Oct 30 06:22:19.158093 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.158075 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:19.188866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.186455 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c"] Oct 30 06:22:19.214279 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.214249 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-f6a022-serving-cert\"" Oct 30 06:22:19.267917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.267882 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjkh2\" (UniqueName: \"kubernetes.io/projected/d45f0929-d6f7-4079-b132-024ed7ea261d-kube-api-access-fjkh2\") pod \"maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:19.268107 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.267937 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45f0929-d6f7-4079-b132-024ed7ea261d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:19.268107 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.268092 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d45f0929-d6f7-4079-b132-024ed7ea261d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:19.268181 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.268140 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45f0929-d6f7-4079-b132-024ed7ea261d-metrics-certs\") pod \"maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:19.277742 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.277716 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:19.372887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.372844 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d45f0929-d6f7-4079-b132-024ed7ea261d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:19.373742 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.373717 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45f0929-d6f7-4079-b132-024ed7ea261d-metrics-certs\") pod \"maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:19.375551 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.375503 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjkh2\" (UniqueName: \"kubernetes.io/projected/d45f0929-d6f7-4079-b132-024ed7ea261d-kube-api-access-fjkh2\") pod \"maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:19.379250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.375932 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45f0929-d6f7-4079-b132-024ed7ea261d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:19.379250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.376729 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45f0929-d6f7-4079-b132-024ed7ea261d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:19.379250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.377195 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d45f0929-d6f7-4079-b132-024ed7ea261d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:19.379250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.378775 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45f0929-d6f7-4079-b132-024ed7ea261d-metrics-certs\") pod \"maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:19.399040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.399010 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjkh2\" (UniqueName: \"kubernetes.io/projected/d45f0929-d6f7-4079-b132-024ed7ea261d-kube-api-access-fjkh2\") pod \"maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:19.424795 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.424716 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:19.472086 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.472047 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:19.782163 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.782132 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c"] Oct 30 06:22:19.784179 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:22:19.784150 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45f0929_d6f7_4079_b132_024ed7ea261d.slice/crio-12701d8ba723601b46887fed21a92628bb5e43da13c56567cc25cdecf67ffa5e WatchSource:0}: Error finding container 12701d8ba723601b46887fed21a92628bb5e43da13c56567cc25cdecf67ffa5e: Status 404 returned error can't find the container with id 12701d8ba723601b46887fed21a92628bb5e43da13c56567cc25cdecf67ffa5e Oct 30 06:22:19.785588 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:19.785570 6999 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:22:20.342512 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:20.342478 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:20.451145 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:20.451021 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" event={"ID":"d45f0929-d6f7-4079-b132-024ed7ea261d","Type":"ContainerStarted","Data":"4c25214225429ec498c86d54f17c9af3b8725919b798d5368e0313041f582819"} Oct 30 06:22:20.451145 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:20.451120 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" event={"ID":"d45f0929-d6f7-4079-b132-024ed7ea261d","Type":"ContainerStarted","Data":"12701d8ba723601b46887fed21a92628bb5e43da13c56567cc25cdecf67ffa5e"} Oct 30 06:22:20.451577 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:20.451547 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:20.492952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:20.492919 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:20.500544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:20.500490 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" podStartSLOduration=1.096656309 podStartE2EDuration="1.500472497s" totalImagesPullingTime="403.816188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:20.497906821 +0000 UTC m=+1707.006926730" watchObservedRunningTime="2026-10-30 06:22:20.500472497 +0000 UTC m=+1707.009492388" Oct 30 06:22:21.319363 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:21.319330 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:21.910086 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:21.910051 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:22.073533 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:22.073484 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:23.415913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:23.415876 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:23.696339 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:23.696258 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:23.760729 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:23.760554 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26"] Oct 30 06:22:23.761037 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:23.760787 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" podUID="d39b2299-705b-4969-8669-1d796017249d" containerName="istio-proxy" containerID="cri-o://0a22023c336bbefbe3445880df6ce7ec6015a472b64e572f4581831198b206e3" gracePeriod=30 Oct 30 06:22:23.847843 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:23.847813 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:24.555203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:24.555164 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:22:24.555696 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:24.555247 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" podUID="d39b2299-705b-4969-8669-1d796017249d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:22:25.306153 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:25.306119 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:26.463643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:26.463604 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:22:27.725572 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:27.725535 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:29.043089 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.043026 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:22:29.211182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.211147 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-istio-envoy\") pod \"d39b2299-705b-4969-8669-1d796017249d\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " Oct 30 06:22:29.211391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.211240 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-workload-certs\") pod \"d39b2299-705b-4969-8669-1d796017249d\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " Oct 30 06:22:29.211391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.211282 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39b2299-705b-4969-8669-1d796017249d-kube-api-access-52qtv\" (UniqueName: \"kubernetes.io/projected/d39b2299-705b-4969-8669-1d796017249d-kube-api-access-52qtv\") pod \"d39b2299-705b-4969-8669-1d796017249d\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " Oct 30 06:22:29.211391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.211336 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-credential-socket\") pod \"d39b2299-705b-4969-8669-1d796017249d\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " Oct 30 06:22:29.211391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.211358 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-workload-socket\") pod \"d39b2299-705b-4969-8669-1d796017249d\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " Oct 30 06:22:29.211628 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.211424 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d39b2299-705b-4969-8669-1d796017249d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39b2299-705b-4969-8669-1d796017249d-istio-podinfo\") pod \"d39b2299-705b-4969-8669-1d796017249d\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " Oct 30 06:22:29.211628 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.211525 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-workload-certs" pod "d39b2299-705b-4969-8669-1d796017249d" (UID: "d39b2299-705b-4969-8669-1d796017249d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:29.211628 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.211537 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d39b2299-705b-4969-8669-1d796017249d-istio-token\" (UniqueName: \"kubernetes.io/projected/d39b2299-705b-4969-8669-1d796017249d-istio-token\") pod \"d39b2299-705b-4969-8669-1d796017249d\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " Oct 30 06:22:29.211781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.211621 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-credential-socket" pod "d39b2299-705b-4969-8669-1d796017249d" (UID: "d39b2299-705b-4969-8669-1d796017249d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:29.211781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.211645 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-istio-data\") pod \"d39b2299-705b-4969-8669-1d796017249d\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " Oct 30 06:22:29.211781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.211694 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d39b2299-705b-4969-8669-1d796017249d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39b2299-705b-4969-8669-1d796017249d-istiod-ca-cert\") pod \"d39b2299-705b-4969-8669-1d796017249d\" (UID: \"d39b2299-705b-4969-8669-1d796017249d\") " Oct 30 06:22:29.211781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.211719 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-workload-socket" pod "d39b2299-705b-4969-8669-1d796017249d" (UID: "d39b2299-705b-4969-8669-1d796017249d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:29.211958 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.211938 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-istio-data" pod "d39b2299-705b-4969-8669-1d796017249d" (UID: "d39b2299-705b-4969-8669-1d796017249d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:29.212414 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.212383 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:29.212414 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.212406 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:29.212567 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.212420 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:29.212567 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.212433 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:29.212567 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.212405 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39b2299-705b-4969-8669-1d796017249d-istiod-ca-cert" pod "d39b2299-705b-4969-8669-1d796017249d" (UID: "d39b2299-705b-4969-8669-1d796017249d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:29.213889 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.213843 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d39b2299-705b-4969-8669-1d796017249d-istio-podinfo" pod "d39b2299-705b-4969-8669-1d796017249d" (UID: "d39b2299-705b-4969-8669-1d796017249d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:29.214018 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.213996 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-istio-envoy" pod "d39b2299-705b-4969-8669-1d796017249d" (UID: "d39b2299-705b-4969-8669-1d796017249d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:29.214181 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.214151 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39b2299-705b-4969-8669-1d796017249d-istio-token" pod "d39b2299-705b-4969-8669-1d796017249d" (UID: "d39b2299-705b-4969-8669-1d796017249d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:29.214332 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.214233 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39b2299-705b-4969-8669-1d796017249d-kube-api-access-52qtv" pod "d39b2299-705b-4969-8669-1d796017249d" (UID: "d39b2299-705b-4969-8669-1d796017249d"). InnerVolumeSpecName "kube-api-access-52qtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:29.314114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.314020 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d39b2299-705b-4969-8669-1d796017249d-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:29.314114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.314057 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d39b2299-705b-4969-8669-1d796017249d-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:29.314114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.314072 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52qtv\" (UniqueName: \"kubernetes.io/projected/d39b2299-705b-4969-8669-1d796017249d-kube-api-access-52qtv\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:29.314114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.314088 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d39b2299-705b-4969-8669-1d796017249d-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:29.314114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.314104 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d39b2299-705b-4969-8669-1d796017249d-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:29.509883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.509851 6999 generic.go:356] "Generic (PLEG): container finished" podID="d39b2299-705b-4969-8669-1d796017249d" containerID="0a22023c336bbefbe3445880df6ce7ec6015a472b64e572f4581831198b206e3" exitCode=0 Oct 30 06:22:29.510043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.509898 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" event={"ID":"d39b2299-705b-4969-8669-1d796017249d","Type":"ContainerDied","Data":"0a22023c336bbefbe3445880df6ce7ec6015a472b64e572f4581831198b206e3"} Oct 30 06:22:29.510043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.509905 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" Oct 30 06:22:29.510043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.509947 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26" event={"ID":"d39b2299-705b-4969-8669-1d796017249d","Type":"ContainerDied","Data":"64eadc650cbfd1ddb27d868b631017f8ca0a0f9f7b32256b5966bb9692ad64bf"} Oct 30 06:22:29.510043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.509972 6999 scope.go:122] "RemoveContainer" containerID="0a22023c336bbefbe3445880df6ce7ec6015a472b64e572f4581831198b206e3" Oct 30 06:22:29.520868 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.520853 6999 scope.go:122] "RemoveContainer" containerID="449e1fb9e7f6c365c15b98c5dc32a38ff728f3fe7a589caed7506294f69db4a7" Oct 30 06:22:29.529925 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.529906 6999 scope.go:122] "RemoveContainer" containerID="0a22023c336bbefbe3445880df6ce7ec6015a472b64e572f4581831198b206e3" Oct 30 06:22:29.530195 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:22:29.530178 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a22023c336bbefbe3445880df6ce7ec6015a472b64e572f4581831198b206e3\": container with ID starting with 0a22023c336bbefbe3445880df6ce7ec6015a472b64e572f4581831198b206e3 not found: ID does not exist" containerID="0a22023c336bbefbe3445880df6ce7ec6015a472b64e572f4581831198b206e3" Oct 30 06:22:29.530269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.530205 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a22023c336bbefbe3445880df6ce7ec6015a472b64e572f4581831198b206e3"} err="failed to get container status \"0a22023c336bbefbe3445880df6ce7ec6015a472b64e572f4581831198b206e3\": rpc error: code = NotFound desc = could not find container \"0a22023c336bbefbe3445880df6ce7ec6015a472b64e572f4581831198b206e3\": container with ID starting with 0a22023c336bbefbe3445880df6ce7ec6015a472b64e572f4581831198b206e3 not found: ID does not exist" Oct 30 06:22:29.530269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.530239 6999 scope.go:122] "RemoveContainer" containerID="449e1fb9e7f6c365c15b98c5dc32a38ff728f3fe7a589caed7506294f69db4a7" Oct 30 06:22:29.530481 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:22:29.530463 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449e1fb9e7f6c365c15b98c5dc32a38ff728f3fe7a589caed7506294f69db4a7\": container with ID starting with 449e1fb9e7f6c365c15b98c5dc32a38ff728f3fe7a589caed7506294f69db4a7 not found: ID does not exist" containerID="449e1fb9e7f6c365c15b98c5dc32a38ff728f3fe7a589caed7506294f69db4a7" Oct 30 06:22:29.530518 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.530486 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449e1fb9e7f6c365c15b98c5dc32a38ff728f3fe7a589caed7506294f69db4a7"} err="failed to get container status \"449e1fb9e7f6c365c15b98c5dc32a38ff728f3fe7a589caed7506294f69db4a7\": rpc error: code = NotFound desc = could not find container \"449e1fb9e7f6c365c15b98c5dc32a38ff728f3fe7a589caed7506294f69db4a7\": container with ID starting with 449e1fb9e7f6c365c15b98c5dc32a38ff728f3fe7a589caed7506294f69db4a7 not found: ID does not exist" Oct 30 06:22:29.549709 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.549685 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:29.561758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.561733 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26"] Oct 30 06:22:29.565499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.565456 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-b4w26"] Oct 30 06:22:29.609927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:29.609892 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d39b2299-705b-4969-8669-1d796017249d" path="/var/lib/kubelet/pods/d39b2299-705b-4969-8669-1d796017249d/volumes" Oct 30 06:22:31.549762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:31.549724 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:32.513208 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:32.513176 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:33.379087 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:33.379002 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:33.895961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:33.895928 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:34.142238 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:22:34.142183 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:22:34.403724 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:34.403692 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:34.563266 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:34.562176 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:35.222132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:35.221992 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:36.568949 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:36.568911 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:37.832000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:37.831970 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:37.987521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:37.987484 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:38.145753 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:38.145722 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:39.478402 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.478349 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74"] Oct 30 06:22:39.478933 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.478665 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" podUID="0ae13585-2214-4d4c-87ec-0e7ee0b57213" containerName="istio-proxy" containerID="cri-o://c82fb6ce0e1eb06c8cb7bcb05e21937c8e8f5b9dbf2fc888cecc99938b903f05" gracePeriod=30 Oct 30 06:22:39.719576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.719542 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:39.864889 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.864851 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq"] Oct 30 06:22:39.865538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.865518 6999 cpu_manager.go:414] "containerMap: removing container" podUID="d39b2299-705b-4969-8669-1d796017249d" containerName="istio-proxy" Oct 30 06:22:39.865654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.865539 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39b2299-705b-4969-8669-1d796017249d" containerName="istio-proxy" Oct 30 06:22:39.865654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.865559 6999 cpu_manager.go:414] "containerMap: removing container" podUID="d39b2299-705b-4969-8669-1d796017249d" containerName="istio-proxy" Oct 30 06:22:39.865654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.865567 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d39b2299-705b-4969-8669-1d796017249d" containerName="istio-proxy" Oct 30 06:22:39.865815 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.865718 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="d39b2299-705b-4969-8669-1d796017249d" containerName="istio-proxy" Oct 30 06:22:39.865815 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.865733 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="d39b2299-705b-4969-8669-1d796017249d" containerName="istio-proxy" Oct 30 06:22:39.869361 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.869342 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" Oct 30 06:22:39.889648 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.889620 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-a035a3-dockercfg-6v5hv\"" Oct 30 06:22:39.889648 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.889647 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-a035a3\"" Oct 30 06:22:39.902381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.902354 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq"] Oct 30 06:22:39.915269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.915242 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:39.948836 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.948803 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn"] Oct 30 06:22:39.952112 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.952088 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b17d593-2ff5-4245-98f7-d63b78885aca-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq\" (UID: \"2b17d593-2ff5-4245-98f7-d63b78885aca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" Oct 30 06:22:39.952290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.952158 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wscr\" (UniqueName: \"kubernetes.io/projected/2b17d593-2ff5-4245-98f7-d63b78885aca-kube-api-access-6wscr\") pod \"payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq\" (UID: \"2b17d593-2ff5-4245-98f7-d63b78885aca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" Oct 30 06:22:39.953502 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.953488 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" Oct 30 06:22:39.978763 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:39.978734 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn"] Oct 30 06:22:40.053794 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.053751 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe251ed-1ad2-4b3f-801a-82fff55b202c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn\" (UID: \"1fe251ed-1ad2-4b3f-801a-82fff55b202c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" Oct 30 06:22:40.053990 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.053816 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwxj9\" (UniqueName: \"kubernetes.io/projected/1fe251ed-1ad2-4b3f-801a-82fff55b202c-kube-api-access-bwxj9\") pod \"payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn\" (UID: \"1fe251ed-1ad2-4b3f-801a-82fff55b202c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" Oct 30 06:22:40.053990 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.053906 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b17d593-2ff5-4245-98f7-d63b78885aca-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq\" (UID: \"2b17d593-2ff5-4245-98f7-d63b78885aca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" Oct 30 06:22:40.053990 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.053967 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wscr\" (UniqueName: \"kubernetes.io/projected/2b17d593-2ff5-4245-98f7-d63b78885aca-kube-api-access-6wscr\") pod \"payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq\" (UID: \"2b17d593-2ff5-4245-98f7-d63b78885aca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" Oct 30 06:22:40.054684 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.054660 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b17d593-2ff5-4245-98f7-d63b78885aca-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq\" (UID: \"2b17d593-2ff5-4245-98f7-d63b78885aca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" Oct 30 06:22:40.077704 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.077677 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wscr\" (UniqueName: \"kubernetes.io/projected/2b17d593-2ff5-4245-98f7-d63b78885aca-kube-api-access-6wscr\") pod \"payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq\" (UID: \"2b17d593-2ff5-4245-98f7-d63b78885aca\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" Oct 30 06:22:40.155259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.155158 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxj9\" (UniqueName: \"kubernetes.io/projected/1fe251ed-1ad2-4b3f-801a-82fff55b202c-kube-api-access-bwxj9\") pod \"payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn\" (UID: \"1fe251ed-1ad2-4b3f-801a-82fff55b202c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" Oct 30 06:22:40.155425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.155404 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe251ed-1ad2-4b3f-801a-82fff55b202c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn\" (UID: \"1fe251ed-1ad2-4b3f-801a-82fff55b202c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" Oct 30 06:22:40.156099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.156077 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe251ed-1ad2-4b3f-801a-82fff55b202c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn\" (UID: \"1fe251ed-1ad2-4b3f-801a-82fff55b202c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" Oct 30 06:22:40.178890 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.178858 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwxj9\" (UniqueName: \"kubernetes.io/projected/1fe251ed-1ad2-4b3f-801a-82fff55b202c-kube-api-access-bwxj9\") pod \"payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn\" (UID: \"1fe251ed-1ad2-4b3f-801a-82fff55b202c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" Oct 30 06:22:40.182783 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.182762 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" Oct 30 06:22:40.191925 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.191903 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:40.270845 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.270808 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" Oct 30 06:22:40.498783 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.498756 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq"] Oct 30 06:22:40.499913 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:22:40.499885 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b17d593_2ff5_4245_98f7_d63b78885aca.slice/crio-614f207538a1c97ae8c39541b2614d466be96d76466770f3485312f7de7b3570 WatchSource:0}: Error finding container 614f207538a1c97ae8c39541b2614d466be96d76466770f3485312f7de7b3570: Status 404 returned error can't find the container with id 614f207538a1c97ae8c39541b2614d466be96d76466770f3485312f7de7b3570 Oct 30 06:22:40.605829 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.600384 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn"] Oct 30 06:22:40.608901 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:22:40.608813 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe251ed_1ad2_4b3f_801a_82fff55b202c.slice/crio-123e07121d7a3c2ec7d0cb5ee1e8c411a878b2c70aa91f8e3f7d9d3b12ccdcb1 WatchSource:0}: Error finding container 123e07121d7a3c2ec7d0cb5ee1e8c411a878b2c70aa91f8e3f7d9d3b12ccdcb1: Status 404 returned error can't find the container with id 123e07121d7a3c2ec7d0cb5ee1e8c411a878b2c70aa91f8e3f7d9d3b12ccdcb1 Oct 30 06:22:40.618043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.618012 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" event={"ID":"2b17d593-2ff5-4245-98f7-d63b78885aca","Type":"ContainerStarted","Data":"091d3b6fc5ddfea80a69f677322569f757a400e1de84b2b86b34cb03665be0fe"} Oct 30 06:22:40.618191 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.618070 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" event={"ID":"2b17d593-2ff5-4245-98f7-d63b78885aca","Type":"ContainerStarted","Data":"614f207538a1c97ae8c39541b2614d466be96d76466770f3485312f7de7b3570"} Oct 30 06:22:40.618191 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.618101 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" Oct 30 06:22:40.619403 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.619377 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" event={"ID":"1fe251ed-1ad2-4b3f-801a-82fff55b202c","Type":"ContainerStarted","Data":"123e07121d7a3c2ec7d0cb5ee1e8c411a878b2c70aa91f8e3f7d9d3b12ccdcb1"} Oct 30 06:22:40.693341 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:40.693297 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" podStartSLOduration=1.693284367 podStartE2EDuration="1.693284367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:40.693000038 +0000 UTC m=+1727.202019925" watchObservedRunningTime="2026-10-30 06:22:40.693284367 +0000 UTC m=+1727.202304254" Oct 30 06:22:41.459858 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:41.459821 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:22:41.460032 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:41.459874 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" podUID="0ae13585-2214-4d4c-87ec-0e7ee0b57213" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:22:41.625892 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:41.625860 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" event={"ID":"1fe251ed-1ad2-4b3f-801a-82fff55b202c","Type":"ContainerStarted","Data":"13312e6893170ac349c025633c28e3d104a9d7a8184d0443858763dd4b93195a"} Oct 30 06:22:41.626379 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:41.625955 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" Oct 30 06:22:41.669044 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:41.668984 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" podStartSLOduration=2.668965449 podStartE2EDuration="2.668965449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:41.667708507 +0000 UTC m=+1728.176728396" watchObservedRunningTime="2026-10-30 06:22:41.668965449 +0000 UTC m=+1728.177985337" Oct 30 06:22:42.460787 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:42.460756 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:44.284585 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.283664 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:44.654950 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.654911 6999 generic.go:356] "Generic (PLEG): container finished" podID="0ae13585-2214-4d4c-87ec-0e7ee0b57213" containerID="c82fb6ce0e1eb06c8cb7bcb05e21937c8e8f5b9dbf2fc888cecc99938b903f05" exitCode=0 Oct 30 06:22:44.655126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.654980 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" event={"ID":"0ae13585-2214-4d4c-87ec-0e7ee0b57213","Type":"ContainerDied","Data":"c82fb6ce0e1eb06c8cb7bcb05e21937c8e8f5b9dbf2fc888cecc99938b903f05"} Oct 30 06:22:44.655126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.655043 6999 scope.go:122] "RemoveContainer" containerID="0d7af5f97642789db4d0ead75e21e35538038446d20d52c0654a80161e93cbae" Oct 30 06:22:44.764178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.764153 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:22:44.916723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.916688 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-credential-socket\") pod \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " Oct 30 06:22:44.916965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.916775 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-envoy\") pod \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " Oct 30 06:22:44.916965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.916870 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-token\" (UniqueName: \"kubernetes.io/projected/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-token\") pod \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " Oct 30 06:22:44.916965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.916900 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-podinfo\") pod \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " Oct 30 06:22:44.916965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.916932 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae13585-2214-4d4c-87ec-0e7ee0b57213-kube-api-access-l7htm\" (UniqueName: \"kubernetes.io/projected/0ae13585-2214-4d4c-87ec-0e7ee0b57213-kube-api-access-l7htm\") pod \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " Oct 30 06:22:44.916965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.916963 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-workload-socket\") pod \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " Oct 30 06:22:44.917301 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.917013 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-credential-socket" pod "0ae13585-2214-4d4c-87ec-0e7ee0b57213" (UID: "0ae13585-2214-4d4c-87ec-0e7ee0b57213"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:44.917301 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.917028 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-workload-certs\") pod \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " Oct 30 06:22:44.917301 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.917085 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istiod-ca-cert\") pod \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " Oct 30 06:22:44.917301 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.917116 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-data\") pod \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\" (UID: \"0ae13585-2214-4d4c-87ec-0e7ee0b57213\") " Oct 30 06:22:44.917500 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.917455 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-workload-certs" pod "0ae13585-2214-4d4c-87ec-0e7ee0b57213" (UID: "0ae13585-2214-4d4c-87ec-0e7ee0b57213"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:44.917579 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.917549 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istiod-ca-cert" pod "0ae13585-2214-4d4c-87ec-0e7ee0b57213" (UID: "0ae13585-2214-4d4c-87ec-0e7ee0b57213"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:44.917738 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.917715 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-data" pod "0ae13585-2214-4d4c-87ec-0e7ee0b57213" (UID: "0ae13585-2214-4d4c-87ec-0e7ee0b57213"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:44.917837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.917798 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-workload-socket" pod "0ae13585-2214-4d4c-87ec-0e7ee0b57213" (UID: "0ae13585-2214-4d4c-87ec-0e7ee0b57213"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:44.918075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.918056 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:44.918125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.918084 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:44.918125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.918099 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:44.918125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.918114 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:44.918264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.918127 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:44.919455 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.919431 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-podinfo" pod "0ae13585-2214-4d4c-87ec-0e7ee0b57213" (UID: "0ae13585-2214-4d4c-87ec-0e7ee0b57213"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:44.919560 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.919468 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-token" pod "0ae13585-2214-4d4c-87ec-0e7ee0b57213" (UID: "0ae13585-2214-4d4c-87ec-0e7ee0b57213"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:44.919680 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.919664 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-envoy" pod "0ae13585-2214-4d4c-87ec-0e7ee0b57213" (UID: "0ae13585-2214-4d4c-87ec-0e7ee0b57213"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:44.919751 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:44.919731 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae13585-2214-4d4c-87ec-0e7ee0b57213-kube-api-access-l7htm" pod "0ae13585-2214-4d4c-87ec-0e7ee0b57213" (UID: "0ae13585-2214-4d4c-87ec-0e7ee0b57213"). InnerVolumeSpecName "kube-api-access-l7htm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:45.019497 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:45.019462 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:45.019497 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:45.019490 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:45.019497 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:45.019501 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7htm\" (UniqueName: \"kubernetes.io/projected/0ae13585-2214-4d4c-87ec-0e7ee0b57213-kube-api-access-l7htm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:45.019740 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:45.019510 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae13585-2214-4d4c-87ec-0e7ee0b57213-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:22:45.662281 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:45.662238 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" Oct 30 06:22:45.662281 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:45.662252 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74" event={"ID":"0ae13585-2214-4d4c-87ec-0e7ee0b57213","Type":"ContainerDied","Data":"650117f6fd693ec7fe1fe7d8fc347bda42c7a8f0452709fd12c911364881a67d"} Oct 30 06:22:45.662790 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:45.662314 6999 scope.go:122] "RemoveContainer" containerID="c82fb6ce0e1eb06c8cb7bcb05e21937c8e8f5b9dbf2fc888cecc99938b903f05" Oct 30 06:22:45.725910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:45.725872 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74"] Oct 30 06:22:45.737922 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:45.737885 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-ghs74"] Oct 30 06:22:46.090433 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:46.090340 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:46.739899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:46.739873 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9_48f8969a-d9f7-49c0-a00c-f3cd1fe78e08/payload-processing/0.log" Oct 30 06:22:46.944912 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:46.944884 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9_48f8969a-d9f7-49c0-a00c-f3cd1fe78e08/payload-processing/0.log" Oct 30 06:22:47.608693 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:47.608650 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae13585-2214-4d4c-87ec-0e7ee0b57213" path="/var/lib/kubelet/pods/0ae13585-2214-4d4c-87ec-0e7ee0b57213/volumes" Oct 30 06:22:47.962873 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:47.962795 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:49.801671 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.801633 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb"] Oct 30 06:22:49.802292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.802278 6999 cpu_manager.go:414] "containerMap: removing container" podUID="0ae13585-2214-4d4c-87ec-0e7ee0b57213" containerName="istio-proxy" Oct 30 06:22:49.802345 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.802295 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae13585-2214-4d4c-87ec-0e7ee0b57213" containerName="istio-proxy" Oct 30 06:22:49.802423 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.802411 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae13585-2214-4d4c-87ec-0e7ee0b57213" containerName="istio-proxy" Oct 30 06:22:49.802465 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.802437 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae13585-2214-4d4c-87ec-0e7ee0b57213" containerName="istio-proxy" Oct 30 06:22:49.807153 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.807137 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:49.833873 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.833842 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-5a6371-serving-cert\"" Oct 30 06:22:49.837943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.837913 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb"] Oct 30 06:22:49.881290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.881254 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97503283-aa66-4f51-b666-c4eff11abc62-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:49.881486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.881426 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97503283-aa66-4f51-b666-c4eff11abc62-metrics-certs\") pod \"maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:49.881587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.881571 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97503283-aa66-4f51-b666-c4eff11abc62-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:49.881646 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.881594 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgmhx\" (UniqueName: \"kubernetes.io/projected/97503283-aa66-4f51-b666-c4eff11abc62-kube-api-access-dgmhx\") pod \"maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:49.927898 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.927861 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:49.984953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.983131 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97503283-aa66-4f51-b666-c4eff11abc62-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:49.984953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.983284 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97503283-aa66-4f51-b666-c4eff11abc62-metrics-certs\") pod \"maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:49.984953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.983418 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97503283-aa66-4f51-b666-c4eff11abc62-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:49.984953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.983445 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgmhx\" (UniqueName: \"kubernetes.io/projected/97503283-aa66-4f51-b666-c4eff11abc62-kube-api-access-dgmhx\") pod \"maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:49.984953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.984567 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97503283-aa66-4f51-b666-c4eff11abc62-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:49.988069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.988037 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97503283-aa66-4f51-b666-c4eff11abc62-metrics-certs\") pod \"maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:49.988251 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:49.988160 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97503283-aa66-4f51-b666-c4eff11abc62-maas-api-tls\") pod \"maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:50.007105 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:50.007074 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgmhx\" (UniqueName: \"kubernetes.io/projected/97503283-aa66-4f51-b666-c4eff11abc62-kube-api-access-dgmhx\") pod \"maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:50.121267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:50.121208 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:50.127432 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:50.127388 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:50.436835 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:50.436810 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb"] Oct 30 06:22:50.439006 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:22:50.438974 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97503283_aa66_4f51_b666_c4eff11abc62.slice/crio-8f13dfc4a5c54c49230bfe07fd54ece90dc387dddf1235693538b8042be167c3 WatchSource:0}: Error finding container 8f13dfc4a5c54c49230bfe07fd54ece90dc387dddf1235693538b8042be167c3: Status 404 returned error can't find the container with id 8f13dfc4a5c54c49230bfe07fd54ece90dc387dddf1235693538b8042be167c3 Oct 30 06:22:50.700912 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:50.700810 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" event={"ID":"97503283-aa66-4f51-b666-c4eff11abc62","Type":"ContainerStarted","Data":"8f13dfc4a5c54c49230bfe07fd54ece90dc387dddf1235693538b8042be167c3"} Oct 30 06:22:51.214634 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:51.214579 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:51.367794 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:51.367763 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:51.629847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:51.629823 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" Oct 30 06:22:51.738288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:51.738249 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" event={"ID":"97503283-aa66-4f51-b666-c4eff11abc62","Type":"ContainerStarted","Data":"1a0156b950b7de0cd5de0c53f0d41a63b74e9c761d9339758d23d716f2b12332"} Oct 30 06:22:51.738469 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:51.738394 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:51.782632 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:51.782580 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" podStartSLOduration=2.475572808 podStartE2EDuration="2.782564414s" totalImagesPullingTime="306.991606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:51.780255645 +0000 UTC m=+1738.289275534" watchObservedRunningTime="2026-10-30 06:22:51.782564414 +0000 UTC m=+1738.291584300" Oct 30 06:22:51.943865 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:51.943786 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:52.645932 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:52.645902 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" Oct 30 06:22:53.758714 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:53.758675 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:54.783808 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:54.783728 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:55.616186 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:55.616138 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:22:57.751502 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:57.751473 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:22:57.791646 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:57.791615 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n"] Oct 30 06:22:57.791860 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:57.791831 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" podUID="495cb545-9b57-4595-88a3-3197d71ab950" containerName="istio-proxy" containerID="cri-o://7b366af470e311bf94c882f26d3d86de4f69618eee4367207c0ff18b38bcc1a0" gracePeriod=30 Oct 30 06:22:58.090535 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:22:58.090450 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:00.269537 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:00.269502 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:00.498140 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:00.498105 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:01.156405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:01.156378 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:02.058684 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:02.058634 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:02.549240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:02.549175 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:03.069496 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.069431 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:23:03.160031 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.159996 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495cb545-9b57-4595-88a3-3197d71ab950-istio-token\" (UniqueName: \"kubernetes.io/projected/495cb545-9b57-4595-88a3-3197d71ab950-istio-token\") pod \"495cb545-9b57-4595-88a3-3197d71ab950\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " Oct 30 06:23:03.160325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.160053 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-credential-socket\") pod \"495cb545-9b57-4595-88a3-3197d71ab950\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " Oct 30 06:23:03.160325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.160084 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-istio-envoy\") pod \"495cb545-9b57-4595-88a3-3197d71ab950\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " Oct 30 06:23:03.160325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.160134 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-workload-socket\") pod \"495cb545-9b57-4595-88a3-3197d71ab950\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " Oct 30 06:23:03.160325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.160208 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495cb545-9b57-4595-88a3-3197d71ab950-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495cb545-9b57-4595-88a3-3197d71ab950-istiod-ca-cert\") pod \"495cb545-9b57-4595-88a3-3197d71ab950\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " Oct 30 06:23:03.160325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.160290 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/495cb545-9b57-4595-88a3-3197d71ab950-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495cb545-9b57-4595-88a3-3197d71ab950-istio-podinfo\") pod \"495cb545-9b57-4595-88a3-3197d71ab950\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " Oct 30 06:23:03.160611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.160340 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-istio-data\") pod \"495cb545-9b57-4595-88a3-3197d71ab950\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " Oct 30 06:23:03.160611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.160408 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495cb545-9b57-4595-88a3-3197d71ab950-kube-api-access-d5htn\" (UniqueName: \"kubernetes.io/projected/495cb545-9b57-4595-88a3-3197d71ab950-kube-api-access-d5htn\") pod \"495cb545-9b57-4595-88a3-3197d71ab950\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " Oct 30 06:23:03.160611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.160444 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-credential-socket" pod "495cb545-9b57-4595-88a3-3197d71ab950" (UID: "495cb545-9b57-4595-88a3-3197d71ab950"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:03.160611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.160480 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-workload-certs\") pod \"495cb545-9b57-4595-88a3-3197d71ab950\" (UID: \"495cb545-9b57-4595-88a3-3197d71ab950\") " Oct 30 06:23:03.160818 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.160666 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-istio-data" pod "495cb545-9b57-4595-88a3-3197d71ab950" (UID: "495cb545-9b57-4595-88a3-3197d71ab950"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:03.160924 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.160903 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-workload-certs" pod "495cb545-9b57-4595-88a3-3197d71ab950" (UID: "495cb545-9b57-4595-88a3-3197d71ab950"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:03.161428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.161059 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-workload-socket" pod "495cb545-9b57-4595-88a3-3197d71ab950" (UID: "495cb545-9b57-4595-88a3-3197d71ab950"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:03.161428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.161118 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.161428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.161137 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.161428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.161185 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.161707 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.161451 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495cb545-9b57-4595-88a3-3197d71ab950-istiod-ca-cert" pod "495cb545-9b57-4595-88a3-3197d71ab950" (UID: "495cb545-9b57-4595-88a3-3197d71ab950"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:03.163626 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.163599 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/495cb545-9b57-4595-88a3-3197d71ab950-istio-podinfo" pod "495cb545-9b57-4595-88a3-3197d71ab950" (UID: "495cb545-9b57-4595-88a3-3197d71ab950"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:03.163732 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.163648 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495cb545-9b57-4595-88a3-3197d71ab950-istio-token" pod "495cb545-9b57-4595-88a3-3197d71ab950" (UID: "495cb545-9b57-4595-88a3-3197d71ab950"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:03.163781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.163752 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495cb545-9b57-4595-88a3-3197d71ab950-kube-api-access-d5htn" pod "495cb545-9b57-4595-88a3-3197d71ab950" (UID: "495cb545-9b57-4595-88a3-3197d71ab950"). InnerVolumeSpecName "kube-api-access-d5htn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:03.164298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.164272 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-istio-envoy" pod "495cb545-9b57-4595-88a3-3197d71ab950" (UID: "495cb545-9b57-4595-88a3-3197d71ab950"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:03.262608 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.262573 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495cb545-9b57-4595-88a3-3197d71ab950-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.262608 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.262603 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.262608 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.262612 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495cb545-9b57-4595-88a3-3197d71ab950-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.262857 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.262621 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495cb545-9b57-4595-88a3-3197d71ab950-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.262857 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.262631 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495cb545-9b57-4595-88a3-3197d71ab950-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.262857 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.262639 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5htn\" (UniqueName: \"kubernetes.io/projected/495cb545-9b57-4595-88a3-3197d71ab950-kube-api-access-d5htn\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.814331 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.814296 6999 generic.go:356] "Generic (PLEG): container finished" podID="495cb545-9b57-4595-88a3-3197d71ab950" containerID="7b366af470e311bf94c882f26d3d86de4f69618eee4367207c0ff18b38bcc1a0" exitCode=0 Oct 30 06:23:03.814515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.814350 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" Oct 30 06:23:03.814515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.814377 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" event={"ID":"495cb545-9b57-4595-88a3-3197d71ab950","Type":"ContainerDied","Data":"7b366af470e311bf94c882f26d3d86de4f69618eee4367207c0ff18b38bcc1a0"} Oct 30 06:23:03.814515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.814431 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n" event={"ID":"495cb545-9b57-4595-88a3-3197d71ab950","Type":"ContainerDied","Data":"49a8706c289153bc6c8d46ee403eab844dea4da845fd45af3c8f750c0fb59a47"} Oct 30 06:23:03.814515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.814457 6999 scope.go:122] "RemoveContainer" containerID="7b366af470e311bf94c882f26d3d86de4f69618eee4367207c0ff18b38bcc1a0" Oct 30 06:23:03.824302 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.824281 6999 scope.go:122] "RemoveContainer" containerID="7b366af470e311bf94c882f26d3d86de4f69618eee4367207c0ff18b38bcc1a0" Oct 30 06:23:03.824596 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:23:03.824575 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b366af470e311bf94c882f26d3d86de4f69618eee4367207c0ff18b38bcc1a0\": container with ID starting with 7b366af470e311bf94c882f26d3d86de4f69618eee4367207c0ff18b38bcc1a0 not found: ID does not exist" containerID="7b366af470e311bf94c882f26d3d86de4f69618eee4367207c0ff18b38bcc1a0" Oct 30 06:23:03.824651 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.824606 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b366af470e311bf94c882f26d3d86de4f69618eee4367207c0ff18b38bcc1a0"} err="failed to get container status \"7b366af470e311bf94c882f26d3d86de4f69618eee4367207c0ff18b38bcc1a0\": rpc error: code = NotFound desc = could not find container \"7b366af470e311bf94c882f26d3d86de4f69618eee4367207c0ff18b38bcc1a0\": container with ID starting with 7b366af470e311bf94c882f26d3d86de4f69618eee4367207c0ff18b38bcc1a0 not found: ID does not exist" Oct 30 06:23:03.837422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.837398 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:03.869910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.869826 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n"] Oct 30 06:23:03.881277 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:03.881238 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-pb87n"] Oct 30 06:23:04.142354 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:23:04.142256 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:23:05.556492 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.556456 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp"] Oct 30 06:23:05.557181 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.557160 6999 cpu_manager.go:414] "containerMap: removing container" podUID="0ae13585-2214-4d4c-87ec-0e7ee0b57213" containerName="istio-proxy" Oct 30 06:23:05.557336 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.557184 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae13585-2214-4d4c-87ec-0e7ee0b57213" containerName="istio-proxy" Oct 30 06:23:05.557336 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.557239 6999 cpu_manager.go:414] "containerMap: removing container" podUID="495cb545-9b57-4595-88a3-3197d71ab950" containerName="istio-proxy" Oct 30 06:23:05.557336 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.557248 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495cb545-9b57-4595-88a3-3197d71ab950" containerName="istio-proxy" Oct 30 06:23:05.557508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.557396 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="495cb545-9b57-4595-88a3-3197d71ab950" containerName="istio-proxy" Oct 30 06:23:05.562035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.562021 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:05.582340 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.582299 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp"] Oct 30 06:23:05.591000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.590966 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-a035a3-serving-cert\"" Oct 30 06:23:05.609767 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.609732 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495cb545-9b57-4595-88a3-3197d71ab950" path="/var/lib/kubelet/pods/495cb545-9b57-4595-88a3-3197d71ab950/volumes" Oct 30 06:23:05.680764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.680732 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:05.694486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.694454 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/add12d67-676a-422f-9dbf-f51027b1ce22-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:05.694655 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.694502 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/add12d67-676a-422f-9dbf-f51027b1ce22-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:05.694723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.694664 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z62d4\" (UniqueName: \"kubernetes.io/projected/add12d67-676a-422f-9dbf-f51027b1ce22-kube-api-access-z62d4\") pod \"maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:05.694723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.694703 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/add12d67-676a-422f-9dbf-f51027b1ce22-metrics-certs\") pod \"maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:05.796264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.796196 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/add12d67-676a-422f-9dbf-f51027b1ce22-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:05.796493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.796286 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/add12d67-676a-422f-9dbf-f51027b1ce22-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:05.796493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.796357 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z62d4\" (UniqueName: \"kubernetes.io/projected/add12d67-676a-422f-9dbf-f51027b1ce22-kube-api-access-z62d4\") pod \"maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:05.796493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.796377 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/add12d67-676a-422f-9dbf-f51027b1ce22-metrics-certs\") pod \"maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:05.796980 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.796950 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/add12d67-676a-422f-9dbf-f51027b1ce22-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:05.798905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.798884 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/add12d67-676a-422f-9dbf-f51027b1ce22-maas-api-tls\") pod \"maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:05.799010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.798925 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/add12d67-676a-422f-9dbf-f51027b1ce22-metrics-certs\") pod \"maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:05.820799 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.820701 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z62d4\" (UniqueName: \"kubernetes.io/projected/add12d67-676a-422f-9dbf-f51027b1ce22-kube-api-access-z62d4\") pod \"maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:05.822295 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.822188 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:05.876282 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:05.876248 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:06.187234 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:06.187179 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp"] Oct 30 06:23:06.189228 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:23:06.189189 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd12d67_676a_422f_9dbf_f51027b1ce22.slice/crio-8987b31bbfead150ebdb5decae081492e6c9fda6df4d2990f245b9beefb639c1 WatchSource:0}: Error finding container 8987b31bbfead150ebdb5decae081492e6c9fda6df4d2990f245b9beefb639c1: Status 404 returned error can't find the container with id 8987b31bbfead150ebdb5decae081492e6c9fda6df4d2990f245b9beefb639c1 Oct 30 06:23:06.838820 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:06.838721 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" event={"ID":"add12d67-676a-422f-9dbf-f51027b1ce22","Type":"ContainerStarted","Data":"ad5d7853e7cac852f212017c3afdd43ae86eb8dd7c306d0f28e12013c237e88e"} Oct 30 06:23:06.839520 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:06.839493 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:06.839637 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:06.839532 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" event={"ID":"add12d67-676a-422f-9dbf-f51027b1ce22","Type":"ContainerStarted","Data":"8987b31bbfead150ebdb5decae081492e6c9fda6df4d2990f245b9beefb639c1"} Oct 30 06:23:06.884379 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:06.884320 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" podStartSLOduration=1.5673593559999999 podStartE2EDuration="1.884302934s" totalImagesPullingTime="316.943578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:06.882383312 +0000 UTC m=+1753.391403208" watchObservedRunningTime="2026-10-30 06:23:06.884302934 +0000 UTC m=+1753.393322869" Oct 30 06:23:07.234350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.234314 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s"] Oct 30 06:23:07.234596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.234565 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" podUID="f0dd6233-fe41-4694-b950-2db40572657d" containerName="maas-api" containerID="cri-o://7c64a379b1b7451c9164c39243d982058ee5bfea3308de7d6eb472be7ed5e2a5" gracePeriod=30 Oct 30 06:23:07.337720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.337660 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:07.485611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.485525 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:07.530433 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.530402 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:23:07.624296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.624266 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0dd6233-fe41-4694-b950-2db40572657d-kube-api-access-gtlzh\" (UniqueName: \"kubernetes.io/projected/f0dd6233-fe41-4694-b950-2db40572657d-kube-api-access-gtlzh\") pod \"f0dd6233-fe41-4694-b950-2db40572657d\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " Oct 30 06:23:07.624480 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.624393 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0dd6233-fe41-4694-b950-2db40572657d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0dd6233-fe41-4694-b950-2db40572657d-openshift-service-ca\") pod \"f0dd6233-fe41-4694-b950-2db40572657d\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " Oct 30 06:23:07.624480 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.624424 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0dd6233-fe41-4694-b950-2db40572657d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0dd6233-fe41-4694-b950-2db40572657d-maas-api-tls\") pod \"f0dd6233-fe41-4694-b950-2db40572657d\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " Oct 30 06:23:07.624553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.624541 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0dd6233-fe41-4694-b950-2db40572657d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0dd6233-fe41-4694-b950-2db40572657d-metrics-certs\") pod \"f0dd6233-fe41-4694-b950-2db40572657d\" (UID: \"f0dd6233-fe41-4694-b950-2db40572657d\") " Oct 30 06:23:07.624762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.624739 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0dd6233-fe41-4694-b950-2db40572657d-openshift-service-ca" pod "f0dd6233-fe41-4694-b950-2db40572657d" (UID: "f0dd6233-fe41-4694-b950-2db40572657d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:07.624986 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.624966 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0dd6233-fe41-4694-b950-2db40572657d-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:07.626690 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.626657 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0dd6233-fe41-4694-b950-2db40572657d-metrics-certs" pod "f0dd6233-fe41-4694-b950-2db40572657d" (UID: "f0dd6233-fe41-4694-b950-2db40572657d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:07.627202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.627172 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0dd6233-fe41-4694-b950-2db40572657d-maas-api-tls" pod "f0dd6233-fe41-4694-b950-2db40572657d" (UID: "f0dd6233-fe41-4694-b950-2db40572657d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:07.627202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.627182 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0dd6233-fe41-4694-b950-2db40572657d-kube-api-access-gtlzh" pod "f0dd6233-fe41-4694-b950-2db40572657d" (UID: "f0dd6233-fe41-4694-b950-2db40572657d"). InnerVolumeSpecName "kube-api-access-gtlzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:07.735238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.726949 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtlzh\" (UniqueName: \"kubernetes.io/projected/f0dd6233-fe41-4694-b950-2db40572657d-kube-api-access-gtlzh\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:07.735238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.727162 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0dd6233-fe41-4694-b950-2db40572657d-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:07.735238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.727180 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0dd6233-fe41-4694-b950-2db40572657d-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:07.846099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.846010 6999 generic.go:356] "Generic (PLEG): container finished" podID="f0dd6233-fe41-4694-b950-2db40572657d" containerID="7c64a379b1b7451c9164c39243d982058ee5bfea3308de7d6eb472be7ed5e2a5" exitCode=0 Oct 30 06:23:07.846099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.846054 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" Oct 30 06:23:07.846633 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.846096 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" event={"ID":"f0dd6233-fe41-4694-b950-2db40572657d","Type":"ContainerDied","Data":"7c64a379b1b7451c9164c39243d982058ee5bfea3308de7d6eb472be7ed5e2a5"} Oct 30 06:23:07.846633 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.846154 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s" event={"ID":"f0dd6233-fe41-4694-b950-2db40572657d","Type":"ContainerDied","Data":"7e85319c9e199921c4b25ec058811f48b8b642d29f8a05d791ec2109f8cd5f7d"} Oct 30 06:23:07.846633 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.846183 6999 scope.go:122] "RemoveContainer" containerID="7c64a379b1b7451c9164c39243d982058ee5bfea3308de7d6eb472be7ed5e2a5" Oct 30 06:23:07.856522 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.856502 6999 scope.go:122] "RemoveContainer" containerID="7c64a379b1b7451c9164c39243d982058ee5bfea3308de7d6eb472be7ed5e2a5" Oct 30 06:23:07.856790 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:23:07.856771 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c64a379b1b7451c9164c39243d982058ee5bfea3308de7d6eb472be7ed5e2a5\": container with ID starting with 7c64a379b1b7451c9164c39243d982058ee5bfea3308de7d6eb472be7ed5e2a5 not found: ID does not exist" containerID="7c64a379b1b7451c9164c39243d982058ee5bfea3308de7d6eb472be7ed5e2a5" Oct 30 06:23:07.856850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.856799 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c64a379b1b7451c9164c39243d982058ee5bfea3308de7d6eb472be7ed5e2a5"} err="failed to get container status \"7c64a379b1b7451c9164c39243d982058ee5bfea3308de7d6eb472be7ed5e2a5\": rpc error: code = NotFound desc = could not find container \"7c64a379b1b7451c9164c39243d982058ee5bfea3308de7d6eb472be7ed5e2a5\": container with ID starting with 7c64a379b1b7451c9164c39243d982058ee5bfea3308de7d6eb472be7ed5e2a5 not found: ID does not exist" Oct 30 06:23:07.898014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.897981 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s"] Oct 30 06:23:07.905237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:07.905184 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-bbd0e5-59676c67f9-g8n6s"] Oct 30 06:23:08.052517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:08.052484 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:09.320552 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:09.320511 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:09.467295 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:09.467254 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:09.609446 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:09.609420 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0dd6233-fe41-4694-b950-2db40572657d" path="/var/lib/kubelet/pods/f0dd6233-fe41-4694-b950-2db40572657d/volumes" Oct 30 06:23:09.942063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:09.941985 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:10.087866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:10.087827 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:11.851738 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:11.851708 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:12.851799 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:12.851774 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:23:13.288958 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.288924 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk"] Oct 30 06:23:13.289196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.289150 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" podUID="dc6b89be-7d33-4cc9-89de-ceeff364e8a4" containerName="maas-api" containerID="cri-o://e6646c25f15446325691c4d25c505de17bf86518c1106b010d1ac09d02e8693b" gracePeriod=30 Oct 30 06:23:13.383918 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.383875 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:13.592089 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.592064 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:23:13.705568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.705521 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-kube-api-access-sn5fk\" (UniqueName: \"kubernetes.io/projected/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-kube-api-access-sn5fk\") pod \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " Oct 30 06:23:13.705784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.705676 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-openshift-service-ca\") pod \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " Oct 30 06:23:13.705784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.705702 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-maas-api-tls\") pod \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " Oct 30 06:23:13.705784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.705752 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-metrics-certs\") pod \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\" (UID: \"dc6b89be-7d33-4cc9-89de-ceeff364e8a4\") " Oct 30 06:23:13.706115 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.706078 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-openshift-service-ca" pod "dc6b89be-7d33-4cc9-89de-ceeff364e8a4" (UID: "dc6b89be-7d33-4cc9-89de-ceeff364e8a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:13.707002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.706977 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:13.708117 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.708090 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-maas-api-tls" pod "dc6b89be-7d33-4cc9-89de-ceeff364e8a4" (UID: "dc6b89be-7d33-4cc9-89de-ceeff364e8a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:13.708250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.708126 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-kube-api-access-sn5fk" pod "dc6b89be-7d33-4cc9-89de-ceeff364e8a4" (UID: "dc6b89be-7d33-4cc9-89de-ceeff364e8a4"). InnerVolumeSpecName "kube-api-access-sn5fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:13.708250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.708181 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-metrics-certs" pod "dc6b89be-7d33-4cc9-89de-ceeff364e8a4" (UID: "dc6b89be-7d33-4cc9-89de-ceeff364e8a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:13.807961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.807925 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn5fk\" (UniqueName: \"kubernetes.io/projected/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-kube-api-access-sn5fk\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:13.807961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.807958 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:13.807961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.807967 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6b89be-7d33-4cc9-89de-ceeff364e8a4-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:13.882633 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.882596 6999 generic.go:356] "Generic (PLEG): container finished" podID="dc6b89be-7d33-4cc9-89de-ceeff364e8a4" containerID="e6646c25f15446325691c4d25c505de17bf86518c1106b010d1ac09d02e8693b" exitCode=0 Oct 30 06:23:13.883111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.882645 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" Oct 30 06:23:13.883111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.882677 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" event={"ID":"dc6b89be-7d33-4cc9-89de-ceeff364e8a4","Type":"ContainerDied","Data":"e6646c25f15446325691c4d25c505de17bf86518c1106b010d1ac09d02e8693b"} Oct 30 06:23:13.883111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.882736 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk" event={"ID":"dc6b89be-7d33-4cc9-89de-ceeff364e8a4","Type":"ContainerDied","Data":"78305e00d06ce913bdb4e2da4a4f1b5393517635cdbeed01da0b8c49fb359ab6"} Oct 30 06:23:13.883111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.882762 6999 scope.go:122] "RemoveContainer" containerID="e6646c25f15446325691c4d25c505de17bf86518c1106b010d1ac09d02e8693b" Oct 30 06:23:13.893437 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.893414 6999 scope.go:122] "RemoveContainer" containerID="e6646c25f15446325691c4d25c505de17bf86518c1106b010d1ac09d02e8693b" Oct 30 06:23:13.893744 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:23:13.893720 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6646c25f15446325691c4d25c505de17bf86518c1106b010d1ac09d02e8693b\": container with ID starting with e6646c25f15446325691c4d25c505de17bf86518c1106b010d1ac09d02e8693b not found: ID does not exist" containerID="e6646c25f15446325691c4d25c505de17bf86518c1106b010d1ac09d02e8693b" Oct 30 06:23:13.893813 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.893759 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6646c25f15446325691c4d25c505de17bf86518c1106b010d1ac09d02e8693b"} err="failed to get container status \"e6646c25f15446325691c4d25c505de17bf86518c1106b010d1ac09d02e8693b\": rpc error: code = NotFound desc = could not find container \"e6646c25f15446325691c4d25c505de17bf86518c1106b010d1ac09d02e8693b\": container with ID starting with e6646c25f15446325691c4d25c505de17bf86518c1106b010d1ac09d02e8693b not found: ID does not exist" Oct 30 06:23:13.913849 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.913816 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:13.935310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.935277 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk"] Oct 30 06:23:13.944166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:13.944140 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-520bc4-6f55ffdcc9-6hlhk"] Oct 30 06:23:14.060978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:14.060949 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:14.202815 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:14.202726 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:14.673757 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:14.673726 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:15.618549 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:15.618512 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6b89be-7d33-4cc9-89de-ceeff364e8a4" path="/var/lib/kubelet/pods/dc6b89be-7d33-4cc9-89de-ceeff364e8a4/volumes" Oct 30 06:23:16.069048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:16.068931 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:17.588041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:17.587996 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:17.747815 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:17.747771 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:17.953127 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:17.953094 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:18.110584 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:18.110540 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:20.043901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:20.043870 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:21.519761 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:21.519730 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:22.417102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:22.417072 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:23.230197 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:23.230163 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:24.663594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:24.663558 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:26.602283 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:26.602244 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:27.619798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.619762 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc"] Oct 30 06:23:27.620675 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.620651 6999 cpu_manager.go:414] "containerMap: removing container" podUID="f0dd6233-fe41-4694-b950-2db40572657d" containerName="maas-api" Oct 30 06:23:27.620828 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.620678 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0dd6233-fe41-4694-b950-2db40572657d" containerName="maas-api" Oct 30 06:23:27.620828 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.620714 6999 cpu_manager.go:414] "containerMap: removing container" podUID="dc6b89be-7d33-4cc9-89de-ceeff364e8a4" containerName="maas-api" Oct 30 06:23:27.620828 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.620722 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6b89be-7d33-4cc9-89de-ceeff364e8a4" containerName="maas-api" Oct 30 06:23:27.620998 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.620908 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0dd6233-fe41-4694-b950-2db40572657d" containerName="maas-api" Oct 30 06:23:27.620998 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.620928 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6b89be-7d33-4cc9-89de-ceeff364e8a4" containerName="maas-api" Oct 30 06:23:27.624946 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.624924 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.654774 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.654744 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-dfa2ae\"/\"kube-root-ca.crt\"" Oct 30 06:23:27.654938 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.654783 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-dfa2ae\"/\"openshift-service-ca.crt\"" Oct 30 06:23:27.655183 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.655160 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-dfa2ae\"/\"praxis-route-dfa2ae-kserve-self-signed-certs\"" Oct 30 06:23:27.655316 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.655232 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-dfa2ae\"/\"default-dockercfg-bz9dh\"" Oct 30 06:23:27.658981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.658960 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc"] Oct 30 06:23:27.678121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.677916 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-model-cache\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.678121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.678039 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-home\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.678121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.678106 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-tmp-dir\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.678423 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.678158 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52465882-affe-4707-8a97-10c1b5405857-tls-certs\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.678546 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.678518 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4ng2\" (UniqueName: \"kubernetes.io/projected/52465882-affe-4707-8a97-10c1b5405857-kube-api-access-f4ng2\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.678627 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.678609 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-dshm\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.706310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.706273 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:27.779793 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.779758 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-model-cache\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.779793 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.779793 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-home\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.780072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.779816 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-tmp-dir\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.780072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.779853 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52465882-affe-4707-8a97-10c1b5405857-tls-certs\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.780072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.780001 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4ng2\" (UniqueName: \"kubernetes.io/projected/52465882-affe-4707-8a97-10c1b5405857-kube-api-access-f4ng2\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.780072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.780058 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-dshm\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.780326 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.780265 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-tmp-dir\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.780386 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.780327 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-home\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.780386 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.780370 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-model-cache\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.782555 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.782533 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-dshm\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.782979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.782957 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52465882-affe-4707-8a97-10c1b5405857-tls-certs\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.803977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.803951 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4ng2\" (UniqueName: \"kubernetes.io/projected/52465882-affe-4707-8a97-10c1b5405857-kube-api-access-f4ng2\") pod \"praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:27.937382 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:27.937286 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:28.251117 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:28.251094 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc"] Oct 30 06:23:28.253182 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:23:28.253156 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52465882_affe_4707_8a97_10c1b5405857.slice/crio-7d71a6d4feccc69b00a322ddcd924f93b775a15995125873f5979ea0072af221 WatchSource:0}: Error finding container 7d71a6d4feccc69b00a322ddcd924f93b775a15995125873f5979ea0072af221: Status 404 returned error can't find the container with id 7d71a6d4feccc69b00a322ddcd924f93b775a15995125873f5979ea0072af221 Oct 30 06:23:28.716683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:28.716650 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:28.985867 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:28.985777 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" event={"ID":"52465882-affe-4707-8a97-10c1b5405857","Type":"ContainerStarted","Data":"91bbac6c18fcb393cc8cf57cdd88fcfe6c0079fe4f0fc82c457486d4ed75bc7f"} Oct 30 06:23:28.985867 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:28.985825 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" event={"ID":"52465882-affe-4707-8a97-10c1b5405857","Type":"ContainerStarted","Data":"7d71a6d4feccc69b00a322ddcd924f93b775a15995125873f5979ea0072af221"} Oct 30 06:23:29.029472 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:29.029419 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" podStartSLOduration=2.02940428 podStartE2EDuration="2.02940428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:29.026687544 +0000 UTC m=+1775.535707422" watchObservedRunningTime="2026-10-30 06:23:29.02940428 +0000 UTC m=+1775.538424180" Oct 30 06:23:29.128162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:29.128130 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z"] Oct 30 06:23:29.268807 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:29.268730 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:29.419290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:29.419248 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:29.938068 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:29.938038 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:30.589965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:30.589926 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:31.464674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:31.464646 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:32.331258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:32.328070 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:32.588905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:32.588835 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:32.745233 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:32.745191 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:33.041940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:33.041911 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:33.219288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:33.219258 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:33.363710 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:33.363677 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:33.700020 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:33.699927 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:34.129486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:34.129396 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" podUID="b95450c7-2660-451e-b701-bd5a43637389" containerName="payload-pre-processing" containerID="cri-o://ade620236888d137b8728b7fd1ea1d6a3194c9d685ad46e9e268dbabefce46d5" gracePeriod=25 Oct 30 06:23:34.144456 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:23:34.144414 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:23:34.402271 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:34.402245 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" Oct 30 06:23:34.590458 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:34.590425 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b95450c7-2660-451e-b701-bd5a43637389-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b95450c7-2660-451e-b701-bd5a43637389-plugins-config-volume\") pod \"b95450c7-2660-451e-b701-bd5a43637389\" (UID: \"b95450c7-2660-451e-b701-bd5a43637389\") " Oct 30 06:23:34.590629 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:34.590535 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b95450c7-2660-451e-b701-bd5a43637389-kube-api-access-j4jmw\" (UniqueName: \"kubernetes.io/projected/b95450c7-2660-451e-b701-bd5a43637389-kube-api-access-j4jmw\") pod \"b95450c7-2660-451e-b701-bd5a43637389\" (UID: \"b95450c7-2660-451e-b701-bd5a43637389\") " Oct 30 06:23:34.590805 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:34.590785 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b95450c7-2660-451e-b701-bd5a43637389-plugins-config-volume" pod "b95450c7-2660-451e-b701-bd5a43637389" (UID: "b95450c7-2660-451e-b701-bd5a43637389"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:34.590913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:34.590889 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b95450c7-2660-451e-b701-bd5a43637389-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:34.592730 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:34.592708 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95450c7-2660-451e-b701-bd5a43637389-kube-api-access-j4jmw" pod "b95450c7-2660-451e-b701-bd5a43637389" (UID: "b95450c7-2660-451e-b701-bd5a43637389"). InnerVolumeSpecName "kube-api-access-j4jmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:34.692077 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:34.691999 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4jmw\" (UniqueName: \"kubernetes.io/projected/b95450c7-2660-451e-b701-bd5a43637389-kube-api-access-j4jmw\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:35.025144 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:35.025054 6999 generic.go:356] "Generic (PLEG): container finished" podID="b95450c7-2660-451e-b701-bd5a43637389" containerID="ade620236888d137b8728b7fd1ea1d6a3194c9d685ad46e9e268dbabefce46d5" exitCode=0 Oct 30 06:23:35.025144 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:35.025105 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" Oct 30 06:23:35.025666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:35.025138 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" event={"ID":"b95450c7-2660-451e-b701-bd5a43637389","Type":"ContainerDied","Data":"ade620236888d137b8728b7fd1ea1d6a3194c9d685ad46e9e268dbabefce46d5"} Oct 30 06:23:35.025666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:35.025190 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z" event={"ID":"b95450c7-2660-451e-b701-bd5a43637389","Type":"ContainerDied","Data":"f168e51e1ab2f7ea79a6d756611a7e1c05b7bc74f7b68289a29b96c6b175f817"} Oct 30 06:23:35.025666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:35.025230 6999 scope.go:122] "RemoveContainer" containerID="ade620236888d137b8728b7fd1ea1d6a3194c9d685ad46e9e268dbabefce46d5" Oct 30 06:23:35.035585 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:35.035564 6999 scope.go:122] "RemoveContainer" containerID="ade620236888d137b8728b7fd1ea1d6a3194c9d685ad46e9e268dbabefce46d5" Oct 30 06:23:35.035928 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:23:35.035904 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade620236888d137b8728b7fd1ea1d6a3194c9d685ad46e9e268dbabefce46d5\": container with ID starting with ade620236888d137b8728b7fd1ea1d6a3194c9d685ad46e9e268dbabefce46d5 not found: ID does not exist" containerID="ade620236888d137b8728b7fd1ea1d6a3194c9d685ad46e9e268dbabefce46d5" Oct 30 06:23:35.035985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:35.035941 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade620236888d137b8728b7fd1ea1d6a3194c9d685ad46e9e268dbabefce46d5"} err="failed to get container status \"ade620236888d137b8728b7fd1ea1d6a3194c9d685ad46e9e268dbabefce46d5\": rpc error: code = NotFound desc = could not find container \"ade620236888d137b8728b7fd1ea1d6a3194c9d685ad46e9e268dbabefce46d5\": container with ID starting with ade620236888d137b8728b7fd1ea1d6a3194c9d685ad46e9e268dbabefce46d5 not found: ID does not exist" Oct 30 06:23:35.083279 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:35.083251 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z"] Oct 30 06:23:35.095164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:35.095134 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-f6a022-d75d4949f-zcw9z"] Oct 30 06:23:35.125431 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:35.125404 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:35.608804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:35.608768 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b95450c7-2660-451e-b701-bd5a43637389" path="/var/lib/kubelet/pods/b95450c7-2660-451e-b701-bd5a43637389/volumes" Oct 30 06:23:37.169849 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:37.169821 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:37.938511 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:37.938472 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:37.952434 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:37.952406 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:38.058003 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:38.057974 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:23:38.627931 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:38.627895 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn"] Oct 30 06:23:38.738288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:38.734245 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:39.051285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:39.051146 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:39.268058 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:39.268030 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:40.499157 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.499125 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c"] Oct 30 06:23:40.499671 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.499386 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" podUID="d45f0929-d6f7-4079-b132-024ed7ea261d" containerName="maas-api" containerID="cri-o://4c25214225429ec498c86d54f17c9af3b8725919b798d5368e0313041f582819" gracePeriod=30 Oct 30 06:23:40.621837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.621806 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:40.799229 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.798934 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:23:40.866600 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.866569 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d45f0929-d6f7-4079-b132-024ed7ea261d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d45f0929-d6f7-4079-b132-024ed7ea261d-maas-api-tls\") pod \"d45f0929-d6f7-4079-b132-024ed7ea261d\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " Oct 30 06:23:40.866833 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.866811 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d45f0929-d6f7-4079-b132-024ed7ea261d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45f0929-d6f7-4079-b132-024ed7ea261d-openshift-service-ca\") pod \"d45f0929-d6f7-4079-b132-024ed7ea261d\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " Oct 30 06:23:40.866900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.866855 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d45f0929-d6f7-4079-b132-024ed7ea261d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45f0929-d6f7-4079-b132-024ed7ea261d-metrics-certs\") pod \"d45f0929-d6f7-4079-b132-024ed7ea261d\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " Oct 30 06:23:40.866969 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.866943 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45f0929-d6f7-4079-b132-024ed7ea261d-kube-api-access-fjkh2\" (UniqueName: \"kubernetes.io/projected/d45f0929-d6f7-4079-b132-024ed7ea261d-kube-api-access-fjkh2\") pod \"d45f0929-d6f7-4079-b132-024ed7ea261d\" (UID: \"d45f0929-d6f7-4079-b132-024ed7ea261d\") " Oct 30 06:23:40.867298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.867254 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45f0929-d6f7-4079-b132-024ed7ea261d-openshift-service-ca" pod "d45f0929-d6f7-4079-b132-024ed7ea261d" (UID: "d45f0929-d6f7-4079-b132-024ed7ea261d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:40.867602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.867555 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d45f0929-d6f7-4079-b132-024ed7ea261d-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:40.872022 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.869880 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d45f0929-d6f7-4079-b132-024ed7ea261d-metrics-certs" pod "d45f0929-d6f7-4079-b132-024ed7ea261d" (UID: "d45f0929-d6f7-4079-b132-024ed7ea261d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:40.872022 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.869986 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d45f0929-d6f7-4079-b132-024ed7ea261d-maas-api-tls" pod "d45f0929-d6f7-4079-b132-024ed7ea261d" (UID: "d45f0929-d6f7-4079-b132-024ed7ea261d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:40.872536 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.872509 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45f0929-d6f7-4079-b132-024ed7ea261d-kube-api-access-fjkh2" pod "d45f0929-d6f7-4079-b132-024ed7ea261d" (UID: "d45f0929-d6f7-4079-b132-024ed7ea261d"). InnerVolumeSpecName "kube-api-access-fjkh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:40.969198 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.969147 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjkh2\" (UniqueName: \"kubernetes.io/projected/d45f0929-d6f7-4079-b132-024ed7ea261d-kube-api-access-fjkh2\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:40.969198 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.969194 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d45f0929-d6f7-4079-b132-024ed7ea261d-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:40.969198 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:40.969226 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d45f0929-d6f7-4079-b132-024ed7ea261d-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.068885 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:41.068794 6999 generic.go:356] "Generic (PLEG): container finished" podID="d45f0929-d6f7-4079-b132-024ed7ea261d" containerID="4c25214225429ec498c86d54f17c9af3b8725919b798d5368e0313041f582819" exitCode=0 Oct 30 06:23:41.068885 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:41.068838 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" Oct 30 06:23:41.069070 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:41.068840 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" event={"ID":"d45f0929-d6f7-4079-b132-024ed7ea261d","Type":"ContainerDied","Data":"4c25214225429ec498c86d54f17c9af3b8725919b798d5368e0313041f582819"} Oct 30 06:23:41.069070 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:41.068943 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c" event={"ID":"d45f0929-d6f7-4079-b132-024ed7ea261d","Type":"ContainerDied","Data":"12701d8ba723601b46887fed21a92628bb5e43da13c56567cc25cdecf67ffa5e"} Oct 30 06:23:41.069070 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:41.068964 6999 scope.go:122] "RemoveContainer" containerID="4c25214225429ec498c86d54f17c9af3b8725919b798d5368e0313041f582819" Oct 30 06:23:41.079228 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:41.079182 6999 scope.go:122] "RemoveContainer" containerID="4c25214225429ec498c86d54f17c9af3b8725919b798d5368e0313041f582819" Oct 30 06:23:41.079546 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:23:41.079524 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c25214225429ec498c86d54f17c9af3b8725919b798d5368e0313041f582819\": container with ID starting with 4c25214225429ec498c86d54f17c9af3b8725919b798d5368e0313041f582819 not found: ID does not exist" containerID="4c25214225429ec498c86d54f17c9af3b8725919b798d5368e0313041f582819" Oct 30 06:23:41.079645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:41.079555 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c25214225429ec498c86d54f17c9af3b8725919b798d5368e0313041f582819"} err="failed to get container status \"4c25214225429ec498c86d54f17c9af3b8725919b798d5368e0313041f582819\": rpc error: code = NotFound desc = could not find container \"4c25214225429ec498c86d54f17c9af3b8725919b798d5368e0313041f582819\": container with ID starting with 4c25214225429ec498c86d54f17c9af3b8725919b798d5368e0313041f582819 not found: ID does not exist" Oct 30 06:23:41.120603 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:41.120579 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c"] Oct 30 06:23:41.132880 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:41.132857 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-f6a022-749b479bbc-rvq7c"] Oct 30 06:23:41.217411 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:41.217381 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:41.365010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:41.364973 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:41.608452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:41.608420 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d45f0929-d6f7-4079-b132-024ed7ea261d" path="/var/lib/kubelet/pods/d45f0929-d6f7-4079-b132-024ed7ea261d/volumes" Oct 30 06:23:41.886878 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:41.886844 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:43.145193 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:43.145140 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:43.443953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:43.443870 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:45.328810 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:45.328774 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:46.586590 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:46.586563 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:46.745425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:46.745398 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:47.421340 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:47.421310 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:49.254615 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:49.254579 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:49.414090 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:49.414056 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:49.640157 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:49.640127 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:49.801772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:49.801742 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:51.486127 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:51.486091 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:52.511171 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:52.511140 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:52.652904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:52.652874 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:53.330869 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:53.330832 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:53.649690 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:53.649628 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" podUID="165ddac3-4c9c-4712-a5f9-2575ccddae94" containerName="main" containerID="cri-o://6abd5ff97dd3d199b326548a00c037d2f07c58cac7c5f096dbfb98e8643d2498" gracePeriod=45 Oct 30 06:23:53.937504 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:53.937480 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:23:54.047646 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.047605 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-home\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-home\") pod \"165ddac3-4c9c-4712-a5f9-2575ccddae94\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " Oct 30 06:23:54.048325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.048189 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-home" pod "165ddac3-4c9c-4712-a5f9-2575ccddae94" (UID: "165ddac3-4c9c-4712-a5f9-2575ccddae94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:54.048506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.048470 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-dshm\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-dshm\") pod \"165ddac3-4c9c-4712-a5f9-2575ccddae94\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " Oct 30 06:23:54.048639 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.048546 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165ddac3-4c9c-4712-a5f9-2575ccddae94-kube-api-access-jt5jn\" (UniqueName: \"kubernetes.io/projected/165ddac3-4c9c-4712-a5f9-2575ccddae94-kube-api-access-jt5jn\") pod \"165ddac3-4c9c-4712-a5f9-2575ccddae94\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " Oct 30 06:23:54.048639 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.048617 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-tmp-dir\") pod \"165ddac3-4c9c-4712-a5f9-2575ccddae94\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " Oct 30 06:23:54.048768 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.048720 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-model-cache\") pod \"165ddac3-4c9c-4712-a5f9-2575ccddae94\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " Oct 30 06:23:54.048983 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.048956 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-tmp-dir" pod "165ddac3-4c9c-4712-a5f9-2575ccddae94" (UID: "165ddac3-4c9c-4712-a5f9-2575ccddae94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:54.048983 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.048969 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-model-cache" pod "165ddac3-4c9c-4712-a5f9-2575ccddae94" (UID: "165ddac3-4c9c-4712-a5f9-2575ccddae94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:54.049125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.049042 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/165ddac3-4c9c-4712-a5f9-2575ccddae94-tls-certs\" (UniqueName: \"kubernetes.io/secret/165ddac3-4c9c-4712-a5f9-2575ccddae94-tls-certs\") pod \"165ddac3-4c9c-4712-a5f9-2575ccddae94\" (UID: \"165ddac3-4c9c-4712-a5f9-2575ccddae94\") " Oct 30 06:23:54.049634 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.049617 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.049764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.049753 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.049854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.049836 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.051135 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.051113 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-dshm" pod "165ddac3-4c9c-4712-a5f9-2575ccddae94" (UID: "165ddac3-4c9c-4712-a5f9-2575ccddae94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:54.051683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.051664 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/165ddac3-4c9c-4712-a5f9-2575ccddae94-tls-certs" pod "165ddac3-4c9c-4712-a5f9-2575ccddae94" (UID: "165ddac3-4c9c-4712-a5f9-2575ccddae94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:54.051772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.051714 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165ddac3-4c9c-4712-a5f9-2575ccddae94-kube-api-access-jt5jn" pod "165ddac3-4c9c-4712-a5f9-2575ccddae94" (UID: "165ddac3-4c9c-4712-a5f9-2575ccddae94"). InnerVolumeSpecName "kube-api-access-jt5jn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:54.148652 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.148615 6999 generic.go:356] "Generic (PLEG): container finished" podID="165ddac3-4c9c-4712-a5f9-2575ccddae94" containerID="6abd5ff97dd3d199b326548a00c037d2f07c58cac7c5f096dbfb98e8643d2498" exitCode=0 Oct 30 06:23:54.148854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.148658 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" Oct 30 06:23:54.148854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.148692 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" event={"ID":"165ddac3-4c9c-4712-a5f9-2575ccddae94","Type":"ContainerDied","Data":"6abd5ff97dd3d199b326548a00c037d2f07c58cac7c5f096dbfb98e8643d2498"} Oct 30 06:23:54.148854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.148736 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn" event={"ID":"165ddac3-4c9c-4712-a5f9-2575ccddae94","Type":"ContainerDied","Data":"52348e135d66ad1952087f4afadf7bd31f6b9381c917f1940fd7d509a9640084"} Oct 30 06:23:54.148854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.148754 6999 scope.go:122] "RemoveContainer" containerID="6abd5ff97dd3d199b326548a00c037d2f07c58cac7c5f096dbfb98e8643d2498" Oct 30 06:23:54.150506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.150484 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/165ddac3-4c9c-4712-a5f9-2575ccddae94-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.150631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.150510 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/165ddac3-4c9c-4712-a5f9-2575ccddae94-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.150631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.150522 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt5jn\" (UniqueName: \"kubernetes.io/projected/165ddac3-4c9c-4712-a5f9-2575ccddae94-kube-api-access-jt5jn\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.159992 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.159946 6999 scope.go:122] "RemoveContainer" containerID="6abd5ff97dd3d199b326548a00c037d2f07c58cac7c5f096dbfb98e8643d2498" Oct 30 06:23:54.160250 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:23:54.160228 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6abd5ff97dd3d199b326548a00c037d2f07c58cac7c5f096dbfb98e8643d2498\": container with ID starting with 6abd5ff97dd3d199b326548a00c037d2f07c58cac7c5f096dbfb98e8643d2498 not found: ID does not exist" containerID="6abd5ff97dd3d199b326548a00c037d2f07c58cac7c5f096dbfb98e8643d2498" Oct 30 06:23:54.160352 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.160255 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6abd5ff97dd3d199b326548a00c037d2f07c58cac7c5f096dbfb98e8643d2498"} err="failed to get container status \"6abd5ff97dd3d199b326548a00c037d2f07c58cac7c5f096dbfb98e8643d2498\": rpc error: code = NotFound desc = could not find container \"6abd5ff97dd3d199b326548a00c037d2f07c58cac7c5f096dbfb98e8643d2498\": container with ID starting with 6abd5ff97dd3d199b326548a00c037d2f07c58cac7c5f096dbfb98e8643d2498 not found: ID does not exist" Oct 30 06:23:54.200584 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.200553 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn"] Oct 30 06:23:54.204303 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.204276 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-532f3b/ipp-route-532f3b-kserve-56cf479545-lwxsn"] Oct 30 06:23:54.805357 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.805251 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb"] Oct 30 06:23:54.805952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.805504 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" podUID="05415885-73d0-40f6-a332-e4dd345f55d2" containerName="maas-api" containerID="cri-o://ecd9b9f0c3603cf4e5bdcca8a4e0628ab3a5ca213f6179538d590919a2ef58d0" gracePeriod=30 Oct 30 06:23:54.942260 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:54.942206 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:55.079978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.079955 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:23:55.159558 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.159512 6999 generic.go:356] "Generic (PLEG): container finished" podID="05415885-73d0-40f6-a332-e4dd345f55d2" containerID="ecd9b9f0c3603cf4e5bdcca8a4e0628ab3a5ca213f6179538d590919a2ef58d0" exitCode=0 Oct 30 06:23:55.159758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.159564 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" event={"ID":"05415885-73d0-40f6-a332-e4dd345f55d2","Type":"ContainerDied","Data":"ecd9b9f0c3603cf4e5bdcca8a4e0628ab3a5ca213f6179538d590919a2ef58d0"} Oct 30 06:23:55.159758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.159616 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" event={"ID":"05415885-73d0-40f6-a332-e4dd345f55d2","Type":"ContainerDied","Data":"328cf0bbdbd938d66ed750dcd3087d958f56d4f011a32b39b7a274cc7adfd42f"} Oct 30 06:23:55.159758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.159575 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb" Oct 30 06:23:55.159758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.159645 6999 scope.go:122] "RemoveContainer" containerID="ecd9b9f0c3603cf4e5bdcca8a4e0628ab3a5ca213f6179538d590919a2ef58d0" Oct 30 06:23:55.169904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.169885 6999 scope.go:122] "RemoveContainer" containerID="ecd9b9f0c3603cf4e5bdcca8a4e0628ab3a5ca213f6179538d590919a2ef58d0" Oct 30 06:23:55.170301 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:23:55.170275 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecd9b9f0c3603cf4e5bdcca8a4e0628ab3a5ca213f6179538d590919a2ef58d0\": container with ID starting with ecd9b9f0c3603cf4e5bdcca8a4e0628ab3a5ca213f6179538d590919a2ef58d0 not found: ID does not exist" containerID="ecd9b9f0c3603cf4e5bdcca8a4e0628ab3a5ca213f6179538d590919a2ef58d0" Oct 30 06:23:55.170388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.170314 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05415885-73d0-40f6-a332-e4dd345f55d2-kube-api-access-b7hqj\" (UniqueName: \"kubernetes.io/projected/05415885-73d0-40f6-a332-e4dd345f55d2-kube-api-access-b7hqj\") pod \"05415885-73d0-40f6-a332-e4dd345f55d2\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " Oct 30 06:23:55.170462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.170311 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecd9b9f0c3603cf4e5bdcca8a4e0628ab3a5ca213f6179538d590919a2ef58d0"} err="failed to get container status \"ecd9b9f0c3603cf4e5bdcca8a4e0628ab3a5ca213f6179538d590919a2ef58d0\": rpc error: code = NotFound desc = could not find container \"ecd9b9f0c3603cf4e5bdcca8a4e0628ab3a5ca213f6179538d590919a2ef58d0\": container with ID starting with ecd9b9f0c3603cf4e5bdcca8a4e0628ab3a5ca213f6179538d590919a2ef58d0 not found: ID does not exist" Oct 30 06:23:55.170462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.170421 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-metrics-certs\") pod \"05415885-73d0-40f6-a332-e4dd345f55d2\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " Oct 30 06:23:55.170591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.170507 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-maas-api-tls\") pod \"05415885-73d0-40f6-a332-e4dd345f55d2\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " Oct 30 06:23:55.170591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.170563 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05415885-73d0-40f6-a332-e4dd345f55d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05415885-73d0-40f6-a332-e4dd345f55d2-openshift-service-ca\") pod \"05415885-73d0-40f6-a332-e4dd345f55d2\" (UID: \"05415885-73d0-40f6-a332-e4dd345f55d2\") " Oct 30 06:23:55.171227 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.171181 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05415885-73d0-40f6-a332-e4dd345f55d2-openshift-service-ca" pod "05415885-73d0-40f6-a332-e4dd345f55d2" (UID: "05415885-73d0-40f6-a332-e4dd345f55d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:55.172737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.172715 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05415885-73d0-40f6-a332-e4dd345f55d2-kube-api-access-b7hqj" pod "05415885-73d0-40f6-a332-e4dd345f55d2" (UID: "05415885-73d0-40f6-a332-e4dd345f55d2"). InnerVolumeSpecName "kube-api-access-b7hqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:55.173023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.172998 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-metrics-certs" pod "05415885-73d0-40f6-a332-e4dd345f55d2" (UID: "05415885-73d0-40f6-a332-e4dd345f55d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:55.173142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.173091 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-maas-api-tls" pod "05415885-73d0-40f6-a332-e4dd345f55d2" (UID: "05415885-73d0-40f6-a332-e4dd345f55d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:55.272109 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.272067 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.272109 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.272109 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05415885-73d0-40f6-a332-e4dd345f55d2-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.272425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.272124 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05415885-73d0-40f6-a332-e4dd345f55d2-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.272425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.272138 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7hqj\" (UniqueName: \"kubernetes.io/projected/05415885-73d0-40f6-a332-e4dd345f55d2-kube-api-access-b7hqj\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.508981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.508945 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb"] Oct 30 06:23:55.516413 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.516376 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:55.521471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.521447 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-4dd310-9fc49cdcd-5cgkb"] Oct 30 06:23:55.609579 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.609537 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05415885-73d0-40f6-a332-e4dd345f55d2" path="/var/lib/kubelet/pods/05415885-73d0-40f6-a332-e4dd345f55d2/volumes" Oct 30 06:23:55.610062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.610039 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="165ddac3-4c9c-4712-a5f9-2575ccddae94" path="/var/lib/kubelet/pods/165ddac3-4c9c-4712-a5f9-2575ccddae94/volumes" Oct 30 06:23:55.662737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:55.662705 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:56.196807 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:56.196781 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:57.617192 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:57.617157 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:23:59.433752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:23:59.433699 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:00.620959 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:00.620925 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:01.222743 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:01.222712 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:01.359300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:01.359265 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:02.674114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:02.674087 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:02.916365 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:02.916333 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:03.084093 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:03.084003 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:03.233457 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:03.233417 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:03.528577 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:03.528500 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:04.142136 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:24:04.142100 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:24:04.836966 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:04.836933 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:05.928913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:05.928881 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:06.125372 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:06.125343 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:06.748321 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:06.748279 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:08.677419 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:08.677361 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:10.704063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:10.704025 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6"] Oct 30 06:24:10.704623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:10.704314 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" podUID="2806ba44-9515-4f41-a30f-f3aa24fe9aaf" containerName="maas-api" containerID="cri-o://59566472b9fb54877c15ecf912ae2d23e9fe3ef75af79dc40d2eeb09ffec366e" gracePeriod=30 Oct 30 06:24:10.801657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:10.801624 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:10.996539 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:10.996511 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:24:11.069412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.069377 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-maas-api-tls\") pod \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " Oct 30 06:24:11.069622 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.069528 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-metrics-certs\") pod \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " Oct 30 06:24:11.069622 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.069589 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-openshift-service-ca\") pod \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " Oct 30 06:24:11.069751 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.069641 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-kube-api-access-cvk7q\" (UniqueName: \"kubernetes.io/projected/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-kube-api-access-cvk7q\") pod \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\" (UID: \"2806ba44-9515-4f41-a30f-f3aa24fe9aaf\") " Oct 30 06:24:11.070263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.070202 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-openshift-service-ca" pod "2806ba44-9515-4f41-a30f-f3aa24fe9aaf" (UID: "2806ba44-9515-4f41-a30f-f3aa24fe9aaf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:11.072347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.072304 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-maas-api-tls" pod "2806ba44-9515-4f41-a30f-f3aa24fe9aaf" (UID: "2806ba44-9515-4f41-a30f-f3aa24fe9aaf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:11.072792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.072750 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-kube-api-access-cvk7q" pod "2806ba44-9515-4f41-a30f-f3aa24fe9aaf" (UID: "2806ba44-9515-4f41-a30f-f3aa24fe9aaf"). InnerVolumeSpecName "kube-api-access-cvk7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:11.072994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.072954 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-metrics-certs" pod "2806ba44-9515-4f41-a30f-f3aa24fe9aaf" (UID: "2806ba44-9515-4f41-a30f-f3aa24fe9aaf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:11.172288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.172248 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:11.172288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.172285 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:11.172545 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.172299 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:11.172545 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.172313 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvk7q\" (UniqueName: \"kubernetes.io/projected/2806ba44-9515-4f41-a30f-f3aa24fe9aaf-kube-api-access-cvk7q\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:11.268470 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.268373 6999 generic.go:356] "Generic (PLEG): container finished" podID="2806ba44-9515-4f41-a30f-f3aa24fe9aaf" containerID="59566472b9fb54877c15ecf912ae2d23e9fe3ef75af79dc40d2eeb09ffec366e" exitCode=0 Oct 30 06:24:11.268470 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.268420 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" Oct 30 06:24:11.268470 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.268430 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" event={"ID":"2806ba44-9515-4f41-a30f-f3aa24fe9aaf","Type":"ContainerDied","Data":"59566472b9fb54877c15ecf912ae2d23e9fe3ef75af79dc40d2eeb09ffec366e"} Oct 30 06:24:11.268743 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.268486 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6" event={"ID":"2806ba44-9515-4f41-a30f-f3aa24fe9aaf","Type":"ContainerDied","Data":"9049380a145190a61921af8bd2226d3ae5eec1b60e268f4c75490d4ce10e228c"} Oct 30 06:24:11.268743 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.268509 6999 scope.go:122] "RemoveContainer" containerID="59566472b9fb54877c15ecf912ae2d23e9fe3ef75af79dc40d2eeb09ffec366e" Oct 30 06:24:11.279336 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.279321 6999 scope.go:122] "RemoveContainer" containerID="59566472b9fb54877c15ecf912ae2d23e9fe3ef75af79dc40d2eeb09ffec366e" Oct 30 06:24:11.279598 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:24:11.279582 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59566472b9fb54877c15ecf912ae2d23e9fe3ef75af79dc40d2eeb09ffec366e\": container with ID starting with 59566472b9fb54877c15ecf912ae2d23e9fe3ef75af79dc40d2eeb09ffec366e not found: ID does not exist" containerID="59566472b9fb54877c15ecf912ae2d23e9fe3ef75af79dc40d2eeb09ffec366e" Oct 30 06:24:11.279648 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.279607 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59566472b9fb54877c15ecf912ae2d23e9fe3ef75af79dc40d2eeb09ffec366e"} err="failed to get container status \"59566472b9fb54877c15ecf912ae2d23e9fe3ef75af79dc40d2eeb09ffec366e\": rpc error: code = NotFound desc = could not find container \"59566472b9fb54877c15ecf912ae2d23e9fe3ef75af79dc40d2eeb09ffec366e\": container with ID starting with 59566472b9fb54877c15ecf912ae2d23e9fe3ef75af79dc40d2eeb09ffec366e not found: ID does not exist" Oct 30 06:24:11.321643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.321607 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6"] Oct 30 06:24:11.332504 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.332477 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-93c22e-6576868cbf-268d6"] Oct 30 06:24:11.394405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.394377 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:11.538105 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.538024 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:11.608953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:11.608914 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2806ba44-9515-4f41-a30f-f3aa24fe9aaf" path="/var/lib/kubelet/pods/2806ba44-9515-4f41-a30f-f3aa24fe9aaf/volumes" Oct 30 06:24:12.126096 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:12.126067 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:13.520866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:13.520832 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:15.380172 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:15.380139 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:15.549878 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:15.549839 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:17.255663 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:17.255613 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:17.401322 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:17.401290 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:19.215575 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:19.215547 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:19.350107 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:19.350063 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:19.682996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:19.682961 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:21.149864 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:21.149830 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:22.939620 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:22.939540 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:24.161587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:24.161545 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:24.313567 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:24.313533 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:24.509516 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:24.509433 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:24.866193 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:24.866154 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:26.898103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:26.898070 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:28.900927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:28.900898 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:31.486803 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:31.486771 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:33.341361 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:33.341326 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:34.142328 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:24:34.142289 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:24:35.164175 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:35.164127 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:36.181006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:36.180964 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn"] Oct 30 06:24:36.208738 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:36.208706 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq"] Oct 30 06:24:36.351795 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:36.351764 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:36.492652 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:36.492567 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:37.043421 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:37.043391 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:38.848876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:38.848843 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:39.860721 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.860660 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt"] Oct 30 06:24:39.861370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.861354 6999 cpu_manager.go:414] "containerMap: removing container" podUID="d45f0929-d6f7-4079-b132-024ed7ea261d" containerName="maas-api" Oct 30 06:24:39.861438 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.861372 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d45f0929-d6f7-4079-b132-024ed7ea261d" containerName="maas-api" Oct 30 06:24:39.861438 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.861387 6999 cpu_manager.go:414] "containerMap: removing container" podUID="05415885-73d0-40f6-a332-e4dd345f55d2" containerName="maas-api" Oct 30 06:24:39.861438 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.861392 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05415885-73d0-40f6-a332-e4dd345f55d2" containerName="maas-api" Oct 30 06:24:39.861438 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.861404 6999 cpu_manager.go:414] "containerMap: removing container" podUID="2806ba44-9515-4f41-a30f-f3aa24fe9aaf" containerName="maas-api" Oct 30 06:24:39.861438 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.861409 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2806ba44-9515-4f41-a30f-f3aa24fe9aaf" containerName="maas-api" Oct 30 06:24:39.861438 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.861426 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b95450c7-2660-451e-b701-bd5a43637389" containerName="payload-pre-processing" Oct 30 06:24:39.861438 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.861430 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b95450c7-2660-451e-b701-bd5a43637389" containerName="payload-pre-processing" Oct 30 06:24:39.861676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.861446 6999 cpu_manager.go:414] "containerMap: removing container" podUID="165ddac3-4c9c-4712-a5f9-2575ccddae94" containerName="main" Oct 30 06:24:39.861676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.861451 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165ddac3-4c9c-4712-a5f9-2575ccddae94" containerName="main" Oct 30 06:24:39.861676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.861541 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="d45f0929-d6f7-4079-b132-024ed7ea261d" containerName="maas-api" Oct 30 06:24:39.861676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.861552 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="05415885-73d0-40f6-a332-e4dd345f55d2" containerName="maas-api" Oct 30 06:24:39.861676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.861558 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="2806ba44-9515-4f41-a30f-f3aa24fe9aaf" containerName="maas-api" Oct 30 06:24:39.861676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.861565 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b95450c7-2660-451e-b701-bd5a43637389" containerName="payload-pre-processing" Oct 30 06:24:39.861676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.861573 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="165ddac3-4c9c-4712-a5f9-2575ccddae94" containerName="main" Oct 30 06:24:39.865059 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.865044 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:39.889768 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.889730 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt"] Oct 30 06:24:39.925464 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.925430 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65f0e55-4e85-4f37-a572-29d7e6635fec-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:39.925641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.925472 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nwww\" (UniqueName: \"kubernetes.io/projected/b65f0e55-4e85-4f37-a572-29d7e6635fec-kube-api-access-7nwww\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:39.925641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.925504 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:39.925641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.925623 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:39.925781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.925680 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:39.925781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.925700 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:39.925781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.925768 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:39.925883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.925827 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:39.925883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.925877 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:39.967857 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:39.967827 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8d44dd-openshift-default-dockercfg-nzqwp\"" Oct 30 06:24:40.027626 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.027591 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nwww\" (UniqueName: \"kubernetes.io/projected/b65f0e55-4e85-4f37-a572-29d7e6635fec-kube-api-access-7nwww\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.027836 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.027635 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.027836 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.027660 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.027836 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.027706 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.027836 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.027724 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.027836 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.027774 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.027836 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.027812 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.028174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.027882 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.028174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.027964 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65f0e55-4e85-4f37-a572-29d7e6635fec-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.028174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.028097 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.028337 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.028181 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.028337 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.028277 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.028455 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.028378 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.028723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.028707 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65f0e55-4e85-4f37-a572-29d7e6635fec-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.030312 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.030294 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.030458 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.030440 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.051206 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.051161 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.051423 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.051388 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nwww\" (UniqueName: \"kubernetes.io/projected/b65f0e55-4e85-4f37-a572-29d7e6635fec-kube-api-access-7nwww\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.179943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.179857 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:40.503088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.503065 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt"] Oct 30 06:24:40.505256 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:24:40.505229 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb65f0e55_4e85_4f37_a572_29d7e6635fec.slice/crio-6e8c9e92466c73edddf2ae93398ac49327e434121b8005b1462c54e3f9fb2e2a WatchSource:0}: Error finding container 6e8c9e92466c73edddf2ae93398ac49327e434121b8005b1462c54e3f9fb2e2a: Status 404 returned error can't find the container with id 6e8c9e92466c73edddf2ae93398ac49327e434121b8005b1462c54e3f9fb2e2a Oct 30 06:24:40.507594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.507559 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:40.507666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.507628 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:40.507666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.507660 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:40.656307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:40.656271 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:41.181303 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.181258 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" podUID="1fe251ed-1ad2-4b3f-801a-82fff55b202c" containerName="payload-processing" containerID="cri-o://13312e6893170ac349c025633c28e3d104a9d7a8184d0443858763dd4b93195a" gracePeriod=25 Oct 30 06:24:41.209351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.209320 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" podUID="2b17d593-2ff5-4245-98f7-d63b78885aca" containerName="payload-pre-processing" containerID="cri-o://091d3b6fc5ddfea80a69f677322569f757a400e1de84b2b86b34cb03665be0fe" gracePeriod=25 Oct 30 06:24:41.481187 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.481158 6999 generic.go:356] "Generic (PLEG): container finished" podID="1fe251ed-1ad2-4b3f-801a-82fff55b202c" containerID="13312e6893170ac349c025633c28e3d104a9d7a8184d0443858763dd4b93195a" exitCode=0 Oct 30 06:24:41.481484 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.481454 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" event={"ID":"1fe251ed-1ad2-4b3f-801a-82fff55b202c","Type":"ContainerDied","Data":"13312e6893170ac349c025633c28e3d104a9d7a8184d0443858763dd4b93195a"} Oct 30 06:24:41.483667 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.483590 6999 generic.go:356] "Generic (PLEG): container finished" podID="2b17d593-2ff5-4245-98f7-d63b78885aca" containerID="091d3b6fc5ddfea80a69f677322569f757a400e1de84b2b86b34cb03665be0fe" exitCode=0 Oct 30 06:24:41.483761 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.483701 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" event={"ID":"2b17d593-2ff5-4245-98f7-d63b78885aca","Type":"ContainerDied","Data":"091d3b6fc5ddfea80a69f677322569f757a400e1de84b2b86b34cb03665be0fe"} Oct 30 06:24:41.485558 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.485537 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" event={"ID":"b65f0e55-4e85-4f37-a572-29d7e6635fec","Type":"ContainerStarted","Data":"4578f7edb272ea8dfdfbcb11154caa3d53ed846668d6a1939e131cc38a977734"} Oct 30 06:24:41.485645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.485574 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" event={"ID":"b65f0e55-4e85-4f37-a572-29d7e6635fec","Type":"ContainerStarted","Data":"6e8c9e92466c73edddf2ae93398ac49327e434121b8005b1462c54e3f9fb2e2a"} Oct 30 06:24:41.506842 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.506775 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" Oct 30 06:24:41.570844 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.570792 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" podStartSLOduration=2.570772055 podStartE2EDuration="2.570772055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:41.558711522 +0000 UTC m=+1848.067731433" watchObservedRunningTime="2026-10-30 06:24:41.570772055 +0000 UTC m=+1848.079791943" Oct 30 06:24:41.587940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.587889 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" Oct 30 06:24:41.670146 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.670114 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b17d593-2ff5-4245-98f7-d63b78885aca-kube-api-access-6wscr\" (UniqueName: \"kubernetes.io/projected/2b17d593-2ff5-4245-98f7-d63b78885aca-kube-api-access-6wscr\") pod \"2b17d593-2ff5-4245-98f7-d63b78885aca\" (UID: \"2b17d593-2ff5-4245-98f7-d63b78885aca\") " Oct 30 06:24:41.670388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.670204 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b17d593-2ff5-4245-98f7-d63b78885aca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b17d593-2ff5-4245-98f7-d63b78885aca-plugins-config-volume\") pod \"2b17d593-2ff5-4245-98f7-d63b78885aca\" (UID: \"2b17d593-2ff5-4245-98f7-d63b78885aca\") " Oct 30 06:24:41.670459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.670439 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe251ed-1ad2-4b3f-801a-82fff55b202c-kube-api-access-bwxj9\" (UniqueName: \"kubernetes.io/projected/1fe251ed-1ad2-4b3f-801a-82fff55b202c-kube-api-access-bwxj9\") pod \"1fe251ed-1ad2-4b3f-801a-82fff55b202c\" (UID: \"1fe251ed-1ad2-4b3f-801a-82fff55b202c\") " Oct 30 06:24:41.670528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.670512 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe251ed-1ad2-4b3f-801a-82fff55b202c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe251ed-1ad2-4b3f-801a-82fff55b202c-plugins-config-volume\") pod \"1fe251ed-1ad2-4b3f-801a-82fff55b202c\" (UID: \"1fe251ed-1ad2-4b3f-801a-82fff55b202c\") " Oct 30 06:24:41.670808 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.670785 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b17d593-2ff5-4245-98f7-d63b78885aca-plugins-config-volume" pod "2b17d593-2ff5-4245-98f7-d63b78885aca" (UID: "2b17d593-2ff5-4245-98f7-d63b78885aca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:41.671012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.670994 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b17d593-2ff5-4245-98f7-d63b78885aca-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:41.671392 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.671324 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe251ed-1ad2-4b3f-801a-82fff55b202c-plugins-config-volume" pod "1fe251ed-1ad2-4b3f-801a-82fff55b202c" (UID: "1fe251ed-1ad2-4b3f-801a-82fff55b202c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:41.673942 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.673921 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe251ed-1ad2-4b3f-801a-82fff55b202c-kube-api-access-bwxj9" pod "1fe251ed-1ad2-4b3f-801a-82fff55b202c" (UID: "1fe251ed-1ad2-4b3f-801a-82fff55b202c"). InnerVolumeSpecName "kube-api-access-bwxj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:41.674046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.674030 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b17d593-2ff5-4245-98f7-d63b78885aca-kube-api-access-6wscr" pod "2b17d593-2ff5-4245-98f7-d63b78885aca" (UID: "2b17d593-2ff5-4245-98f7-d63b78885aca"). InnerVolumeSpecName "kube-api-access-6wscr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:41.772312 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.772183 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fe251ed-1ad2-4b3f-801a-82fff55b202c-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:41.772312 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.772250 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wscr\" (UniqueName: \"kubernetes.io/projected/2b17d593-2ff5-4245-98f7-d63b78885aca-kube-api-access-6wscr\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:41.772312 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:41.772265 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwxj9\" (UniqueName: \"kubernetes.io/projected/1fe251ed-1ad2-4b3f-801a-82fff55b202c-kube-api-access-bwxj9\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.180198 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:42.180168 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:42.181397 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:42.181375 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:24:42.181802 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:42.181424 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" podUID="b65f0e55-4e85-4f37-a572-29d7e6635fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 06:24:42.492118 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:42.492085 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" Oct 30 06:24:42.492118 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:42.492099 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn" event={"ID":"1fe251ed-1ad2-4b3f-801a-82fff55b202c","Type":"ContainerDied","Data":"123e07121d7a3c2ec7d0cb5ee1e8c411a878b2c70aa91f8e3f7d9d3b12ccdcb1"} Oct 30 06:24:42.492403 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:42.492180 6999 scope.go:122] "RemoveContainer" containerID="13312e6893170ac349c025633c28e3d104a9d7a8184d0443858763dd4b93195a" Oct 30 06:24:42.493884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:42.493867 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" Oct 30 06:24:42.494000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:42.493879 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq" event={"ID":"2b17d593-2ff5-4245-98f7-d63b78885aca","Type":"ContainerDied","Data":"614f207538a1c97ae8c39541b2614d466be96d76466770f3485312f7de7b3570"} Oct 30 06:24:42.503589 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:42.503477 6999 scope.go:122] "RemoveContainer" containerID="091d3b6fc5ddfea80a69f677322569f757a400e1de84b2b86b34cb03665be0fe" Oct 30 06:24:42.559236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:42.559172 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn"] Oct 30 06:24:42.570025 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:42.570006 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a035a3-557b7d66b8-cd2hn"] Oct 30 06:24:42.597123 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:42.597092 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq"] Oct 30 06:24:42.602185 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:42.602160 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a035a3-7cf78c4447-lx4lq"] Oct 30 06:24:42.660047 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:42.660018 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:43.180877 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:43.180831 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:24:43.181069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:43.180901 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" podUID="b65f0e55-4e85-4f37-a572-29d7e6635fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 06:24:43.608719 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:43.608693 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe251ed-1ad2-4b3f-801a-82fff55b202c" path="/var/lib/kubelet/pods/1fe251ed-1ad2-4b3f-801a-82fff55b202c/volumes" Oct 30 06:24:43.609195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:43.609178 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b17d593-2ff5-4245-98f7-d63b78885aca" path="/var/lib/kubelet/pods/2b17d593-2ff5-4245-98f7-d63b78885aca/volumes" Oct 30 06:24:44.180185 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:44.180149 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:24:44.180379 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:44.180203 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" podUID="b65f0e55-4e85-4f37-a572-29d7e6635fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 06:24:44.576703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:44.576626 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:45.180168 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:45.180134 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:24:45.180606 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:45.180200 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" podUID="b65f0e55-4e85-4f37-a572-29d7e6635fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 06:24:46.180179 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:46.180151 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:24:46.180595 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:46.180200 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" podUID="b65f0e55-4e85-4f37-a572-29d7e6635fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 06:24:46.554588 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:46.554455 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:47.180858 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:47.180829 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 06:24:47.181262 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:47.180877 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" podUID="b65f0e55-4e85-4f37-a572-29d7e6635fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 06:24:48.030446 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.030410 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:48.105785 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.105747 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9"] Oct 30 06:24:48.106038 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.106004 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" podUID="48f8969a-d9f7-49c0-a00c-f3cd1fe78e08" containerName="payload-processing" containerID="cri-o://d1f6bfaf646393481209cb0446e8a78d609c19bc1c9c673595ef603b1fb0d5bb" gracePeriod=30 Oct 30 06:24:48.126265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.124983 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj"] Oct 30 06:24:48.126265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.125202 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" podUID="744d5495-1eae-487c-92f2-172f6fb29bcf" containerName="payload-pre-processing" containerID="cri-o://73e4b30066c40c60b8beb8acc0dccb8b4d753982ec460498a86df7f6ebb0ab53" gracePeriod=30 Oct 30 06:24:48.241015 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.239460 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:48.396396 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.396346 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:48.418094 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.416675 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:48.418094 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.417945 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:48.420858 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.420814 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:24:48.494851 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.494770 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" Oct 30 06:24:48.575078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.572942 6999 generic.go:356] "Generic (PLEG): container finished" podID="48f8969a-d9f7-49c0-a00c-f3cd1fe78e08" containerID="d1f6bfaf646393481209cb0446e8a78d609c19bc1c9c673595ef603b1fb0d5bb" exitCode=0 Oct 30 06:24:48.575078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.573110 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" event={"ID":"48f8969a-d9f7-49c0-a00c-f3cd1fe78e08","Type":"ContainerDied","Data":"d1f6bfaf646393481209cb0446e8a78d609c19bc1c9c673595ef603b1fb0d5bb"} Oct 30 06:24:48.575078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.573178 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" event={"ID":"48f8969a-d9f7-49c0-a00c-f3cd1fe78e08","Type":"ContainerDied","Data":"e01d753d049cff13ecf8bb7f36bb6499ef6ff8e2d88aae23aeb94c07b6bfe865"} Oct 30 06:24:48.575078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.573231 6999 scope.go:122] "RemoveContainer" containerID="d1f6bfaf646393481209cb0446e8a78d609c19bc1c9c673595ef603b1fb0d5bb" Oct 30 06:24:48.575078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.573423 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9" Oct 30 06:24:48.575078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.574309 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" Oct 30 06:24:48.578231 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.577559 6999 generic.go:356] "Generic (PLEG): container finished" podID="744d5495-1eae-487c-92f2-172f6fb29bcf" containerID="73e4b30066c40c60b8beb8acc0dccb8b4d753982ec460498a86df7f6ebb0ab53" exitCode=0 Oct 30 06:24:48.585446 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.578424 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" event={"ID":"744d5495-1eae-487c-92f2-172f6fb29bcf","Type":"ContainerDied","Data":"73e4b30066c40c60b8beb8acc0dccb8b4d753982ec460498a86df7f6ebb0ab53"} Oct 30 06:24:48.585446 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.578473 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" event={"ID":"744d5495-1eae-487c-92f2-172f6fb29bcf","Type":"ContainerDied","Data":"6b6fad5a925162454383408af7c884e625bedd201bc50c58a7ea922563b98898"} Oct 30 06:24:48.590672 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.589559 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48f8969a-d9f7-49c0-a00c-f3cd1fe78e08-kube-api-access-8746x\" (UniqueName: \"kubernetes.io/projected/48f8969a-d9f7-49c0-a00c-f3cd1fe78e08-kube-api-access-8746x\") pod \"48f8969a-d9f7-49c0-a00c-f3cd1fe78e08\" (UID: \"48f8969a-d9f7-49c0-a00c-f3cd1fe78e08\") " Oct 30 06:24:48.590672 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.589744 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744d5495-1eae-487c-92f2-172f6fb29bcf-kube-api-access-5z5rc\" (UniqueName: \"kubernetes.io/projected/744d5495-1eae-487c-92f2-172f6fb29bcf-kube-api-access-5z5rc\") pod \"744d5495-1eae-487c-92f2-172f6fb29bcf\" (UID: \"744d5495-1eae-487c-92f2-172f6fb29bcf\") " Oct 30 06:24:48.590672 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.589789 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48f8969a-d9f7-49c0-a00c-f3cd1fe78e08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f8969a-d9f7-49c0-a00c-f3cd1fe78e08-plugins-config-volume\") pod \"48f8969a-d9f7-49c0-a00c-f3cd1fe78e08\" (UID: \"48f8969a-d9f7-49c0-a00c-f3cd1fe78e08\") " Oct 30 06:24:48.590672 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.589898 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/744d5495-1eae-487c-92f2-172f6fb29bcf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744d5495-1eae-487c-92f2-172f6fb29bcf-plugins-config-volume\") pod \"744d5495-1eae-487c-92f2-172f6fb29bcf\" (UID: \"744d5495-1eae-487c-92f2-172f6fb29bcf\") " Oct 30 06:24:48.595073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.594629 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f8969a-d9f7-49c0-a00c-f3cd1fe78e08-plugins-config-volume" pod "48f8969a-d9f7-49c0-a00c-f3cd1fe78e08" (UID: "48f8969a-d9f7-49c0-a00c-f3cd1fe78e08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:48.595457 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.595431 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744d5495-1eae-487c-92f2-172f6fb29bcf-plugins-config-volume" pod "744d5495-1eae-487c-92f2-172f6fb29bcf" (UID: "744d5495-1eae-487c-92f2-172f6fb29bcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:48.595638 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.595619 6999 scope.go:122] "RemoveContainer" containerID="d1f6bfaf646393481209cb0446e8a78d609c19bc1c9c673595ef603b1fb0d5bb" Oct 30 06:24:48.596363 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:24:48.596118 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f6bfaf646393481209cb0446e8a78d609c19bc1c9c673595ef603b1fb0d5bb\": container with ID starting with d1f6bfaf646393481209cb0446e8a78d609c19bc1c9c673595ef603b1fb0d5bb not found: ID does not exist" containerID="d1f6bfaf646393481209cb0446e8a78d609c19bc1c9c673595ef603b1fb0d5bb" Oct 30 06:24:48.596363 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.596282 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f6bfaf646393481209cb0446e8a78d609c19bc1c9c673595ef603b1fb0d5bb"} err="failed to get container status \"d1f6bfaf646393481209cb0446e8a78d609c19bc1c9c673595ef603b1fb0d5bb\": rpc error: code = NotFound desc = could not find container \"d1f6bfaf646393481209cb0446e8a78d609c19bc1c9c673595ef603b1fb0d5bb\": container with ID starting with d1f6bfaf646393481209cb0446e8a78d609c19bc1c9c673595ef603b1fb0d5bb not found: ID does not exist" Oct 30 06:24:48.596363 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.596309 6999 scope.go:122] "RemoveContainer" containerID="73e4b30066c40c60b8beb8acc0dccb8b4d753982ec460498a86df7f6ebb0ab53" Oct 30 06:24:48.612478 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.612457 6999 scope.go:122] "RemoveContainer" containerID="73e4b30066c40c60b8beb8acc0dccb8b4d753982ec460498a86df7f6ebb0ab53" Oct 30 06:24:48.612892 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:24:48.612868 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e4b30066c40c60b8beb8acc0dccb8b4d753982ec460498a86df7f6ebb0ab53\": container with ID starting with 73e4b30066c40c60b8beb8acc0dccb8b4d753982ec460498a86df7f6ebb0ab53 not found: ID does not exist" containerID="73e4b30066c40c60b8beb8acc0dccb8b4d753982ec460498a86df7f6ebb0ab53" Oct 30 06:24:48.612991 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.612904 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e4b30066c40c60b8beb8acc0dccb8b4d753982ec460498a86df7f6ebb0ab53"} err="failed to get container status \"73e4b30066c40c60b8beb8acc0dccb8b4d753982ec460498a86df7f6ebb0ab53\": rpc error: code = NotFound desc = could not find container \"73e4b30066c40c60b8beb8acc0dccb8b4d753982ec460498a86df7f6ebb0ab53\": container with ID starting with 73e4b30066c40c60b8beb8acc0dccb8b4d753982ec460498a86df7f6ebb0ab53 not found: ID does not exist" Oct 30 06:24:48.614780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.614741 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744d5495-1eae-487c-92f2-172f6fb29bcf-kube-api-access-5z5rc" pod "744d5495-1eae-487c-92f2-172f6fb29bcf" (UID: "744d5495-1eae-487c-92f2-172f6fb29bcf"). InnerVolumeSpecName "kube-api-access-5z5rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:48.614916 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.614890 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f8969a-d9f7-49c0-a00c-f3cd1fe78e08-kube-api-access-8746x" pod "48f8969a-d9f7-49c0-a00c-f3cd1fe78e08" (UID: "48f8969a-d9f7-49c0-a00c-f3cd1fe78e08"). InnerVolumeSpecName "kube-api-access-8746x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:48.692160 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.692072 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z5rc\" (UniqueName: \"kubernetes.io/projected/744d5495-1eae-487c-92f2-172f6fb29bcf-kube-api-access-5z5rc\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:48.692160 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.692107 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48f8969a-d9f7-49c0-a00c-f3cd1fe78e08-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:48.692160 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.692121 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/744d5495-1eae-487c-92f2-172f6fb29bcf-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:48.692160 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.692134 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8746x\" (UniqueName: \"kubernetes.io/projected/48f8969a-d9f7-49c0-a00c-f3cd1fe78e08-kube-api-access-8746x\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:48.757008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.756972 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:48.963263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.963205 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9"] Oct 30 06:24:48.978669 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:48.978631 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2cf8a-66c6f86c68-7w5m9"] Oct 30 06:24:49.038105 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:49.038069 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:49.190485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:49.190390 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:49.521284 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:49.520731 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:49.616554 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:49.616242 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj" Oct 30 06:24:49.616739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:49.616623 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48f8969a-d9f7-49c0-a00c-f3cd1fe78e08" path="/var/lib/kubelet/pods/48f8969a-d9f7-49c0-a00c-f3cd1fe78e08/volumes" Oct 30 06:24:49.707236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:49.706273 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj"] Oct 30 06:24:49.716759 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:49.707555 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2cf8a-74b9d4596c-2mrrj"] Oct 30 06:24:51.025864 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:51.025831 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:51.609889 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:51.609849 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="744d5495-1eae-487c-92f2-172f6fb29bcf" path="/var/lib/kubelet/pods/744d5495-1eae-487c-92f2-172f6fb29bcf/volumes" Oct 30 06:24:52.424864 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.424829 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9"] Oct 30 06:24:52.425434 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.425419 6999 cpu_manager.go:414] "containerMap: removing container" podUID="1fe251ed-1ad2-4b3f-801a-82fff55b202c" containerName="payload-processing" Oct 30 06:24:52.425480 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.425436 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe251ed-1ad2-4b3f-801a-82fff55b202c" containerName="payload-processing" Oct 30 06:24:52.425480 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.425452 6999 cpu_manager.go:414] "containerMap: removing container" podUID="48f8969a-d9f7-49c0-a00c-f3cd1fe78e08" containerName="payload-processing" Oct 30 06:24:52.425480 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.425458 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48f8969a-d9f7-49c0-a00c-f3cd1fe78e08" containerName="payload-processing" Oct 30 06:24:52.425480 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.425470 6999 cpu_manager.go:414] "containerMap: removing container" podUID="744d5495-1eae-487c-92f2-172f6fb29bcf" containerName="payload-pre-processing" Oct 30 06:24:52.425480 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.425476 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="744d5495-1eae-487c-92f2-172f6fb29bcf" containerName="payload-pre-processing" Oct 30 06:24:52.425636 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.425486 6999 cpu_manager.go:414] "containerMap: removing container" podUID="2b17d593-2ff5-4245-98f7-d63b78885aca" containerName="payload-pre-processing" Oct 30 06:24:52.425636 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.425492 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b17d593-2ff5-4245-98f7-d63b78885aca" containerName="payload-pre-processing" Oct 30 06:24:52.425636 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.425568 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="48f8969a-d9f7-49c0-a00c-f3cd1fe78e08" containerName="payload-processing" Oct 30 06:24:52.425636 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.425578 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="744d5495-1eae-487c-92f2-172f6fb29bcf" containerName="payload-pre-processing" Oct 30 06:24:52.425636 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.425584 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b17d593-2ff5-4245-98f7-d63b78885aca" containerName="payload-pre-processing" Oct 30 06:24:52.425636 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.425597 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe251ed-1ad2-4b3f-801a-82fff55b202c" containerName="payload-processing" Oct 30 06:24:52.429923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.429899 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.457316 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.457281 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.457667 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.457640 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.457971 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.457948 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.458083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.457983 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.458083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.458018 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.458201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.458089 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.458201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.458116 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.458201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.458150 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.458403 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.458251 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw4d2\" (UniqueName: \"kubernetes.io/projected/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-kube-api-access-qw4d2\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.458403 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.458250 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9"] Oct 30 06:24:52.559501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.559458 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw4d2\" (UniqueName: \"kubernetes.io/projected/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-kube-api-access-qw4d2\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.559737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.559537 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.559737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.559569 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.559737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.559640 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.559919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.559730 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.559919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.559792 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.559919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.559875 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.559919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.559902 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.560128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.559933 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.560180 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.560143 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-credential-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.560266 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.560206 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-workload-socket\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.560331 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.560291 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-data\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.560507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.560382 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-workload-certs\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.560608 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.560506 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istiod-ca-cert\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.562396 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.562377 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-envoy\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.562582 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.562562 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-podinfo\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.583994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.583966 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw4d2\" (UniqueName: \"kubernetes.io/projected/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-kube-api-access-qw4d2\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.584083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.583972 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-token\") pod \"e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.630609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.630560 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp"] Oct 30 06:24:52.630785 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.630763 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" podUID="add12d67-676a-422f-9dbf-f51027b1ce22" containerName="maas-api" containerID="cri-o://ad5d7853e7cac852f212017c3afdd43ae86eb8dd7c306d0f28e12013c237e88e" gracePeriod=30 Oct 30 06:24:52.728975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.728855 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:52.745881 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.745841 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:52.847258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.847025 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" podUID="add12d67-676a-422f-9dbf-f51027b1ce22" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.128:8443/health\": dial tcp 10.134.0.128:8443: connect: connection refused" Oct 30 06:24:52.994926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:52.994890 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:24:53.067015 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.066984 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/add12d67-676a-422f-9dbf-f51027b1ce22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/add12d67-676a-422f-9dbf-f51027b1ce22-metrics-certs\") pod \"add12d67-676a-422f-9dbf-f51027b1ce22\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " Oct 30 06:24:53.067241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.067042 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add12d67-676a-422f-9dbf-f51027b1ce22-kube-api-access-z62d4\" (UniqueName: \"kubernetes.io/projected/add12d67-676a-422f-9dbf-f51027b1ce22-kube-api-access-z62d4\") pod \"add12d67-676a-422f-9dbf-f51027b1ce22\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " Oct 30 06:24:53.067241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.067140 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/add12d67-676a-422f-9dbf-f51027b1ce22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/add12d67-676a-422f-9dbf-f51027b1ce22-maas-api-tls\") pod \"add12d67-676a-422f-9dbf-f51027b1ce22\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " Oct 30 06:24:53.067241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.067206 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/add12d67-676a-422f-9dbf-f51027b1ce22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/add12d67-676a-422f-9dbf-f51027b1ce22-openshift-service-ca\") pod \"add12d67-676a-422f-9dbf-f51027b1ce22\" (UID: \"add12d67-676a-422f-9dbf-f51027b1ce22\") " Oct 30 06:24:53.067602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.067570 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add12d67-676a-422f-9dbf-f51027b1ce22-openshift-service-ca" pod "add12d67-676a-422f-9dbf-f51027b1ce22" (UID: "add12d67-676a-422f-9dbf-f51027b1ce22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:53.067859 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.067825 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/add12d67-676a-422f-9dbf-f51027b1ce22-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:53.069480 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.069456 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add12d67-676a-422f-9dbf-f51027b1ce22-metrics-certs" pod "add12d67-676a-422f-9dbf-f51027b1ce22" (UID: "add12d67-676a-422f-9dbf-f51027b1ce22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:53.069597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.069540 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add12d67-676a-422f-9dbf-f51027b1ce22-kube-api-access-z62d4" pod "add12d67-676a-422f-9dbf-f51027b1ce22" (UID: "add12d67-676a-422f-9dbf-f51027b1ce22"). InnerVolumeSpecName "kube-api-access-z62d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:53.069761 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.069746 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add12d67-676a-422f-9dbf-f51027b1ce22-maas-api-tls" pod "add12d67-676a-422f-9dbf-f51027b1ce22" (UID: "add12d67-676a-422f-9dbf-f51027b1ce22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:53.080061 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:24:53.080032 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2e77f4_eed7_46c9_b1cf_c02f7863431c.slice/crio-ce01d2adffdec97c0f42b7a6c302b2f390bcf7b9ac01c00f73d9239dd40456a1 WatchSource:0}: Error finding container ce01d2adffdec97c0f42b7a6c302b2f390bcf7b9ac01c00f73d9239dd40456a1: Status 404 returned error can't find the container with id ce01d2adffdec97c0f42b7a6c302b2f390bcf7b9ac01c00f73d9239dd40456a1 Oct 30 06:24:53.080345 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.080324 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9"] Oct 30 06:24:53.082722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.082685 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:53.082821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.082774 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:53.082888 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.082847 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:24:53.192234 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.192177 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/add12d67-676a-422f-9dbf-f51027b1ce22-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:53.192234 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.192239 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/add12d67-676a-422f-9dbf-f51027b1ce22-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:53.192480 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.192254 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z62d4\" (UniqueName: \"kubernetes.io/projected/add12d67-676a-422f-9dbf-f51027b1ce22-kube-api-access-z62d4\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:24:53.310257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.310108 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:53.462415 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.462370 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:53.639882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.639841 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" event={"ID":"2c2e77f4-eed7-46c9-b1cf-c02f7863431c","Type":"ContainerStarted","Data":"be6c9bc98adc3320234f6585a388301a4d356ac57ee98a45009ce20b1b0fb1e8"} Oct 30 06:24:53.640080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.639907 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" event={"ID":"2c2e77f4-eed7-46c9-b1cf-c02f7863431c","Type":"ContainerStarted","Data":"ce01d2adffdec97c0f42b7a6c302b2f390bcf7b9ac01c00f73d9239dd40456a1"} Oct 30 06:24:53.641187 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.641162 6999 generic.go:356] "Generic (PLEG): container finished" podID="add12d67-676a-422f-9dbf-f51027b1ce22" containerID="ad5d7853e7cac852f212017c3afdd43ae86eb8dd7c306d0f28e12013c237e88e" exitCode=0 Oct 30 06:24:53.641370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.641200 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" Oct 30 06:24:53.641370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.641255 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" event={"ID":"add12d67-676a-422f-9dbf-f51027b1ce22","Type":"ContainerDied","Data":"ad5d7853e7cac852f212017c3afdd43ae86eb8dd7c306d0f28e12013c237e88e"} Oct 30 06:24:53.641370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.641296 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp" event={"ID":"add12d67-676a-422f-9dbf-f51027b1ce22","Type":"ContainerDied","Data":"8987b31bbfead150ebdb5decae081492e6c9fda6df4d2990f245b9beefb639c1"} Oct 30 06:24:53.641370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.641316 6999 scope.go:122] "RemoveContainer" containerID="ad5d7853e7cac852f212017c3afdd43ae86eb8dd7c306d0f28e12013c237e88e" Oct 30 06:24:53.650962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.650940 6999 scope.go:122] "RemoveContainer" containerID="ad5d7853e7cac852f212017c3afdd43ae86eb8dd7c306d0f28e12013c237e88e" Oct 30 06:24:53.651316 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:24:53.651290 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5d7853e7cac852f212017c3afdd43ae86eb8dd7c306d0f28e12013c237e88e\": container with ID starting with ad5d7853e7cac852f212017c3afdd43ae86eb8dd7c306d0f28e12013c237e88e not found: ID does not exist" containerID="ad5d7853e7cac852f212017c3afdd43ae86eb8dd7c306d0f28e12013c237e88e" Oct 30 06:24:53.651397 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.651329 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5d7853e7cac852f212017c3afdd43ae86eb8dd7c306d0f28e12013c237e88e"} err="failed to get container status \"ad5d7853e7cac852f212017c3afdd43ae86eb8dd7c306d0f28e12013c237e88e\": rpc error: code = NotFound desc = could not find container \"ad5d7853e7cac852f212017c3afdd43ae86eb8dd7c306d0f28e12013c237e88e\": container with ID starting with ad5d7853e7cac852f212017c3afdd43ae86eb8dd7c306d0f28e12013c237e88e not found: ID does not exist" Oct 30 06:24:53.686281 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.686205 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" podStartSLOduration=1.686193161 podStartE2EDuration="1.686193161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:53.684523624 +0000 UTC m=+1860.193543512" watchObservedRunningTime="2026-10-30 06:24:53.686193161 +0000 UTC m=+1860.195213104" Oct 30 06:24:53.724075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.724043 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp"] Oct 30 06:24:53.729821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.729793 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-a035a3-dd876b47-ktlhp"] Oct 30 06:24:53.746542 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:53.746507 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:24:54.011923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:54.011875 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:54.763267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:54.761639 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:24:54.763267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:54.761720 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" podUID="2c2e77f4-eed7-46c9-b1cf-c02f7863431c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 06:24:55.487540 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:55.487497 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:55.610814 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:55.610781 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add12d67-676a-422f-9dbf-f51027b1ce22" path="/var/lib/kubelet/pods/add12d67-676a-422f-9dbf-f51027b1ce22/volumes" Oct 30 06:24:55.758299 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:55.757622 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:24:55.758299 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:55.757684 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" podUID="2c2e77f4-eed7-46c9-b1cf-c02f7863431c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 06:24:56.606841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:56.606810 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:56.747638 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:56.747600 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:24:56.747990 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:56.747950 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" podUID="2c2e77f4-eed7-46c9-b1cf-c02f7863431c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 06:24:56.795228 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:56.795177 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:57.170910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:57.170853 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:57.550610 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:57.550475 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:57.747294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:57.747258 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:24:57.748005 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:57.747972 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" podUID="2c2e77f4-eed7-46c9-b1cf-c02f7863431c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 06:24:58.664017 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:58.663967 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:58.755090 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:58.755053 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:24:58.756992 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:58.755749 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" podUID="2c2e77f4-eed7-46c9-b1cf-c02f7863431c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 06:24:59.214955 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:59.214916 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:59.513376 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:59.513285 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:59.660045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:59.660001 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:24:59.747075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:59.747042 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 06:24:59.747292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:24:59.747096 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" podUID="2c2e77f4-eed7-46c9-b1cf-c02f7863431c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 06:25:00.755168 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:00.755119 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:00.852095 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:00.852046 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:25:00.852889 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:00.852819 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:25:00.853731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:00.853710 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:25:01.424425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:01.424390 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:01.579288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:01.579239 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:01.918684 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:01.918646 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:02.548636 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:02.548606 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:02.696451 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:02.696416 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:02.844276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:02.844169 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:03.136984 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:03.136951 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:03.424206 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:03.424109 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:03.607775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:03.607733 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:04.141255 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:25:04.141199 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:25:04.904521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:04.904490 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:05.121982 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:05.121943 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:05.710043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:05.710003 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:07.774758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:07.774728 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:09.609393 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:09.609357 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:11.198919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.198889 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7"] Oct 30 06:25:11.199444 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.199400 6999 cpu_manager.go:414] "containerMap: removing container" podUID="add12d67-676a-422f-9dbf-f51027b1ce22" containerName="maas-api" Oct 30 06:25:11.199444 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.199412 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add12d67-676a-422f-9dbf-f51027b1ce22" containerName="maas-api" Oct 30 06:25:11.200486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.200441 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="add12d67-676a-422f-9dbf-f51027b1ce22" containerName="maas-api" Oct 30 06:25:11.211901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.211873 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" Oct 30 06:25:11.226244 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.226203 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7"] Oct 30 06:25:11.233486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.233466 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:25:11.233614 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.233562 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:25:11.334247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.334181 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3605490-9f6d-4acd-9cba-631cd9d017c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7f5c011-tsmn7\" (UID: \"a3605490-9f6d-4acd-9cba-631cd9d017c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" Oct 30 06:25:11.334422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.334305 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bchj\" (UniqueName: \"kubernetes.io/projected/a3605490-9f6d-4acd-9cba-631cd9d017c2-kube-api-access-2bchj\") pod \"maas-api-revoke-sub-c7f5c011-tsmn7\" (UID: \"a3605490-9f6d-4acd-9cba-631cd9d017c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" Oct 30 06:25:11.334422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.334401 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3605490-9f6d-4acd-9cba-631cd9d017c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7f5c011-tsmn7\" (UID: \"a3605490-9f6d-4acd-9cba-631cd9d017c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" Oct 30 06:25:11.435064 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.435038 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bchj\" (UniqueName: \"kubernetes.io/projected/a3605490-9f6d-4acd-9cba-631cd9d017c2-kube-api-access-2bchj\") pod \"maas-api-revoke-sub-c7f5c011-tsmn7\" (UID: \"a3605490-9f6d-4acd-9cba-631cd9d017c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" Oct 30 06:25:11.435224 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.435137 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3605490-9f6d-4acd-9cba-631cd9d017c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7f5c011-tsmn7\" (UID: \"a3605490-9f6d-4acd-9cba-631cd9d017c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" Oct 30 06:25:11.435283 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.435265 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3605490-9f6d-4acd-9cba-631cd9d017c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7f5c011-tsmn7\" (UID: \"a3605490-9f6d-4acd-9cba-631cd9d017c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" Oct 30 06:25:11.435759 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.435738 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3605490-9f6d-4acd-9cba-631cd9d017c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7f5c011-tsmn7\" (UID: \"a3605490-9f6d-4acd-9cba-631cd9d017c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" Oct 30 06:25:11.437727 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.437704 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3605490-9f6d-4acd-9cba-631cd9d017c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7f5c011-tsmn7\" (UID: \"a3605490-9f6d-4acd-9cba-631cd9d017c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" Oct 30 06:25:11.458926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.458874 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bchj\" (UniqueName: \"kubernetes.io/projected/a3605490-9f6d-4acd-9cba-631cd9d017c2-kube-api-access-2bchj\") pod \"maas-api-revoke-sub-c7f5c011-tsmn7\" (UID: \"a3605490-9f6d-4acd-9cba-631cd9d017c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" Oct 30 06:25:11.531348 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.531324 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" Oct 30 06:25:11.701031 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.701002 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:11.845041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:11.845012 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7"] Oct 30 06:25:11.845746 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:25:11.845722 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3605490_9f6d_4acd_9cba_631cd9d017c2.slice/crio-d19584864a77a62a9bed6453ea1e3aa7005beb4ce21bedb046e5c82e81ad891d WatchSource:0}: Error finding container d19584864a77a62a9bed6453ea1e3aa7005beb4ce21bedb046e5c82e81ad891d: Status 404 returned error can't find the container with id d19584864a77a62a9bed6453ea1e3aa7005beb4ce21bedb046e5c82e81ad891d Oct 30 06:25:12.846370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:12.846341 6999 generic.go:356] "Generic (PLEG): container finished" podID="a3605490-9f6d-4acd-9cba-631cd9d017c2" containerID="67d0b5fb26b92b30fde2debcfe712d8df1c1357e01a6a2db8c5de6d7185d8148" exitCode=0 Oct 30 06:25:12.846772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:12.846421 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" event={"ID":"a3605490-9f6d-4acd-9cba-631cd9d017c2","Type":"ContainerDied","Data":"67d0b5fb26b92b30fde2debcfe712d8df1c1357e01a6a2db8c5de6d7185d8148"} Oct 30 06:25:12.846772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:12.846466 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" event={"ID":"a3605490-9f6d-4acd-9cba-631cd9d017c2","Type":"ContainerStarted","Data":"d19584864a77a62a9bed6453ea1e3aa7005beb4ce21bedb046e5c82e81ad891d"} Oct 30 06:25:13.785686 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:13.785651 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:14.717563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:14.717539 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" Oct 30 06:25:14.854501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:14.854458 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3605490-9f6d-4acd-9cba-631cd9d017c2-kube-api-access-2bchj\" (UniqueName: \"kubernetes.io/projected/a3605490-9f6d-4acd-9cba-631cd9d017c2-kube-api-access-2bchj\") pod \"a3605490-9f6d-4acd-9cba-631cd9d017c2\" (UID: \"a3605490-9f6d-4acd-9cba-631cd9d017c2\") " Oct 30 06:25:14.854693 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:14.854662 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3605490-9f6d-4acd-9cba-631cd9d017c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3605490-9f6d-4acd-9cba-631cd9d017c2-maas-api-openshift-ca\") pod \"a3605490-9f6d-4acd-9cba-631cd9d017c2\" (UID: \"a3605490-9f6d-4acd-9cba-631cd9d017c2\") " Oct 30 06:25:14.854768 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:14.854709 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3605490-9f6d-4acd-9cba-631cd9d017c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3605490-9f6d-4acd-9cba-631cd9d017c2-maas-api-xks-ca\") pod \"a3605490-9f6d-4acd-9cba-631cd9d017c2\" (UID: \"a3605490-9f6d-4acd-9cba-631cd9d017c2\") " Oct 30 06:25:14.855104 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:14.855074 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3605490-9f6d-4acd-9cba-631cd9d017c2-maas-api-openshift-ca" pod "a3605490-9f6d-4acd-9cba-631cd9d017c2" (UID: "a3605490-9f6d-4acd-9cba-631cd9d017c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:14.855250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:14.855229 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3605490-9f6d-4acd-9cba-631cd9d017c2-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:25:14.857613 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:14.857591 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3605490-9f6d-4acd-9cba-631cd9d017c2-kube-api-access-2bchj" pod "a3605490-9f6d-4acd-9cba-631cd9d017c2" (UID: "a3605490-9f6d-4acd-9cba-631cd9d017c2"). InnerVolumeSpecName "kube-api-access-2bchj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:14.857613 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:14.857592 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3605490-9f6d-4acd-9cba-631cd9d017c2-maas-api-xks-ca" pod "a3605490-9f6d-4acd-9cba-631cd9d017c2" (UID: "a3605490-9f6d-4acd-9cba-631cd9d017c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:14.886289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:14.886191 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" event={"ID":"a3605490-9f6d-4acd-9cba-631cd9d017c2","Type":"ContainerDied","Data":"d19584864a77a62a9bed6453ea1e3aa7005beb4ce21bedb046e5c82e81ad891d"} Oct 30 06:25:14.886420 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:14.886301 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d19584864a77a62a9bed6453ea1e3aa7005beb4ce21bedb046e5c82e81ad891d" Oct 30 06:25:14.886420 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:14.886322 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7" Oct 30 06:25:14.956498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:14.956470 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3605490-9f6d-4acd-9cba-631cd9d017c2-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:25:14.956498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:14.956496 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bchj\" (UniqueName: \"kubernetes.io/projected/a3605490-9f6d-4acd-9cba-631cd9d017c2-kube-api-access-2bchj\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:25:16.053798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:16.053766 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:18.008195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.008160 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n"] Oct 30 06:25:18.008729 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.008714 6999 cpu_manager.go:414] "containerMap: removing container" podUID="a3605490-9f6d-4acd-9cba-631cd9d017c2" containerName="revoke-keys" Oct 30 06:25:18.008783 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.008731 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3605490-9f6d-4acd-9cba-631cd9d017c2" containerName="revoke-keys" Oct 30 06:25:18.008868 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.008858 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3605490-9f6d-4acd-9cba-631cd9d017c2" containerName="revoke-keys" Oct 30 06:25:18.012369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.012354 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:18.030652 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.030629 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-f5e55f-serving-cert\"" Oct 30 06:25:18.044847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.044820 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n"] Oct 30 06:25:18.092787 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.092751 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0191799b-7939-49e1-b3b9-496270844a86-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:18.092957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.092898 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0191799b-7939-49e1-b3b9-496270844a86-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:18.092957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.092944 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0191799b-7939-49e1-b3b9-496270844a86-metrics-certs\") pod \"maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:18.093102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.093071 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm7p6\" (UniqueName: \"kubernetes.io/projected/0191799b-7939-49e1-b3b9-496270844a86-kube-api-access-dm7p6\") pod \"maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:18.194455 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.194427 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm7p6\" (UniqueName: \"kubernetes.io/projected/0191799b-7939-49e1-b3b9-496270844a86-kube-api-access-dm7p6\") pod \"maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:18.194605 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.194495 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0191799b-7939-49e1-b3b9-496270844a86-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:18.194605 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.194586 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0191799b-7939-49e1-b3b9-496270844a86-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:18.194690 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.194631 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0191799b-7939-49e1-b3b9-496270844a86-metrics-certs\") pod \"maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:18.195257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.195175 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0191799b-7939-49e1-b3b9-496270844a86-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:18.197313 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.197282 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0191799b-7939-49e1-b3b9-496270844a86-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:18.197432 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.197325 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:18.197432 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.197374 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0191799b-7939-49e1-b3b9-496270844a86-metrics-certs\") pod \"maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:18.225000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.224971 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm7p6\" (UniqueName: \"kubernetes.io/projected/0191799b-7939-49e1-b3b9-496270844a86-kube-api-access-dm7p6\") pod \"maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:18.326542 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.326449 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:18.381486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.380423 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:18.633815 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.633794 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n"] Oct 30 06:25:18.636018 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:25:18.635994 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0191799b_7939_49e1_b3b9_496270844a86.slice/crio-1ac56a160d4f7914779cdb1f6961ca094c55bf3b34614410ff3152fcc8f2a20f WatchSource:0}: Error finding container 1ac56a160d4f7914779cdb1f6961ca094c55bf3b34614410ff3152fcc8f2a20f: Status 404 returned error can't find the container with id 1ac56a160d4f7914779cdb1f6961ca094c55bf3b34614410ff3152fcc8f2a20f Oct 30 06:25:18.910263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:18.910167 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" event={"ID":"0191799b-7939-49e1-b3b9-496270844a86","Type":"ContainerStarted","Data":"1ac56a160d4f7914779cdb1f6961ca094c55bf3b34614410ff3152fcc8f2a20f"} Oct 30 06:25:19.919037 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:19.918997 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" event={"ID":"0191799b-7939-49e1-b3b9-496270844a86","Type":"ContainerStarted","Data":"a9411f19e64162a2a9fa86a414c584f5883c99e16d7e069ed235f601d2c77bb6"} Oct 30 06:25:19.919495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:19.919235 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:19.968926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:19.968878 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" podStartSLOduration=2.561750966 podStartE2EDuration="2.968865064s" totalImagesPullingTime="407.114098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:19.966357296 +0000 UTC m=+1886.475377183" watchObservedRunningTime="2026-10-30 06:25:19.968865064 +0000 UTC m=+1886.477884959" Oct 30 06:25:20.152200 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:20.152157 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:22.112271 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:22.112236 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:24.044468 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:24.044436 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:25.930495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:25.930468 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:25:26.033923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:26.033881 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:27.938107 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:27.938072 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:29.730019 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:29.729986 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:31.528975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:31.528945 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:33.021004 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.020975 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9"] Oct 30 06:25:33.021505 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.021177 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" podUID="9a8acf4a-d7d3-4910-84c6-767d624bf65a" containerName="maas-api" containerID="cri-o://1e5c3f63f7cb97a6b162b41557060563ceb22823aeae853fa10063933f8a5c1b" gracePeriod=30 Oct 30 06:25:33.132684 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.132645 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:33.526279 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.524345 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:25:33.677767 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.677738 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a8acf4a-d7d3-4910-84c6-767d624bf65a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8acf4a-d7d3-4910-84c6-767d624bf65a-metrics-certs\") pod \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " Oct 30 06:25:33.677934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.677914 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8acf4a-d7d3-4910-84c6-767d624bf65a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8acf4a-d7d3-4910-84c6-767d624bf65a-openshift-service-ca\") pod \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " Oct 30 06:25:33.677989 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.677957 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8acf4a-d7d3-4910-84c6-767d624bf65a-kube-api-access-9tbbq\" (UniqueName: \"kubernetes.io/projected/9a8acf4a-d7d3-4910-84c6-767d624bf65a-kube-api-access-9tbbq\") pod \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " Oct 30 06:25:33.678032 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.678021 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a8acf4a-d7d3-4910-84c6-767d624bf65a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a8acf4a-d7d3-4910-84c6-767d624bf65a-maas-api-tls\") pod \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\" (UID: \"9a8acf4a-d7d3-4910-84c6-767d624bf65a\") " Oct 30 06:25:33.678430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.678365 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8acf4a-d7d3-4910-84c6-767d624bf65a-openshift-service-ca" pod "9a8acf4a-d7d3-4910-84c6-767d624bf65a" (UID: "9a8acf4a-d7d3-4910-84c6-767d624bf65a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:33.680000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.679977 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8acf4a-d7d3-4910-84c6-767d624bf65a-metrics-certs" pod "9a8acf4a-d7d3-4910-84c6-767d624bf65a" (UID: "9a8acf4a-d7d3-4910-84c6-767d624bf65a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:33.680458 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.680438 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8acf4a-d7d3-4910-84c6-767d624bf65a-maas-api-tls" pod "9a8acf4a-d7d3-4910-84c6-767d624bf65a" (UID: "9a8acf4a-d7d3-4910-84c6-767d624bf65a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:33.680621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.680604 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8acf4a-d7d3-4910-84c6-767d624bf65a-kube-api-access-9tbbq" pod "9a8acf4a-d7d3-4910-84c6-767d624bf65a" (UID: "9a8acf4a-d7d3-4910-84c6-767d624bf65a"). InnerVolumeSpecName "kube-api-access-9tbbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:33.692154 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.692128 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:33.780122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.780087 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a8acf4a-d7d3-4910-84c6-767d624bf65a-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:25:33.783028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.780114 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a8acf4a-d7d3-4910-84c6-767d624bf65a-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:25:33.783028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.780524 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tbbq\" (UniqueName: \"kubernetes.io/projected/9a8acf4a-d7d3-4910-84c6-767d624bf65a-kube-api-access-9tbbq\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:25:33.783028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.780545 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a8acf4a-d7d3-4910-84c6-767d624bf65a-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:25:33.841716 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:33.841677 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:34.001391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:34.001303 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:34.002659 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:34.002632 6999 generic.go:356] "Generic (PLEG): container finished" podID="9a8acf4a-d7d3-4910-84c6-767d624bf65a" containerID="1e5c3f63f7cb97a6b162b41557060563ceb22823aeae853fa10063933f8a5c1b" exitCode=0 Oct 30 06:25:34.002803 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:34.002690 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" Oct 30 06:25:34.002803 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:34.002720 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" event={"ID":"9a8acf4a-d7d3-4910-84c6-767d624bf65a","Type":"ContainerDied","Data":"1e5c3f63f7cb97a6b162b41557060563ceb22823aeae853fa10063933f8a5c1b"} Oct 30 06:25:34.002803 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:34.002796 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9" event={"ID":"9a8acf4a-d7d3-4910-84c6-767d624bf65a","Type":"ContainerDied","Data":"e4b45fa2f7705a846e3dcb64f0358cd022bbe4d7d7d81218c3a14836c471cf96"} Oct 30 06:25:34.002910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:34.002820 6999 scope.go:122] "RemoveContainer" containerID="1e5c3f63f7cb97a6b162b41557060563ceb22823aeae853fa10063933f8a5c1b" Oct 30 06:25:34.013974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:34.013956 6999 scope.go:122] "RemoveContainer" containerID="1e5c3f63f7cb97a6b162b41557060563ceb22823aeae853fa10063933f8a5c1b" Oct 30 06:25:34.014289 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:25:34.014269 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5c3f63f7cb97a6b162b41557060563ceb22823aeae853fa10063933f8a5c1b\": container with ID starting with 1e5c3f63f7cb97a6b162b41557060563ceb22823aeae853fa10063933f8a5c1b not found: ID does not exist" containerID="1e5c3f63f7cb97a6b162b41557060563ceb22823aeae853fa10063933f8a5c1b" Oct 30 06:25:34.014328 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:34.014298 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5c3f63f7cb97a6b162b41557060563ceb22823aeae853fa10063933f8a5c1b"} err="failed to get container status \"1e5c3f63f7cb97a6b162b41557060563ceb22823aeae853fa10063933f8a5c1b\": rpc error: code = NotFound desc = could not find container \"1e5c3f63f7cb97a6b162b41557060563ceb22823aeae853fa10063933f8a5c1b\": container with ID starting with 1e5c3f63f7cb97a6b162b41557060563ceb22823aeae853fa10063933f8a5c1b not found: ID does not exist" Oct 30 06:25:34.057964 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:34.057944 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9"] Oct 30 06:25:34.070053 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:34.070027 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-dfa2ae-78dbb45d58-bhbf9"] Oct 30 06:25:34.141070 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:25:34.141037 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:25:34.425574 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:34.425547 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:35.610368 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:35.610335 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8acf4a-d7d3-4910-84c6-767d624bf65a" path="/var/lib/kubelet/pods/9a8acf4a-d7d3-4910-84c6-767d624bf65a/volumes" Oct 30 06:25:36.037150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:36.037064 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:38.367729 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:38.367694 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:38.607049 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:38.607016 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:39.389160 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:39.389131 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc"] Oct 30 06:25:39.572925 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:39.572890 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:39.919280 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:39.919249 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:40.055259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.054937 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg"] Oct 30 06:25:40.059267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.055872 6999 cpu_manager.go:414] "containerMap: removing container" podUID="9a8acf4a-d7d3-4910-84c6-767d624bf65a" containerName="maas-api" Oct 30 06:25:40.059267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.055895 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8acf4a-d7d3-4910-84c6-767d624bf65a" containerName="maas-api" Oct 30 06:25:40.059267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.056035 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8acf4a-d7d3-4910-84c6-767d624bf65a" containerName="maas-api" Oct 30 06:25:40.060784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.060760 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.093486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.093458 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg"] Oct 30 06:25:40.157520 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.157463 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.157681 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.157600 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.157681 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.157645 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.157681 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.157663 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.157885 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.157759 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.157885 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.157857 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.157965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.157886 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88d9911-2196-41cd-a9ae-3e73aca123e5-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.157965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.157952 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pkcc\" (UniqueName: \"kubernetes.io/projected/e88d9911-2196-41cd-a9ae-3e73aca123e5-kube-api-access-7pkcc\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.158047 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.158005 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.258744 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.258672 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.258744 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.258731 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.258952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.258789 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.258952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.258833 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.258952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.258858 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.258952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.258904 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.258952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.258934 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.259158 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.259049 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88d9911-2196-41cd-a9ae-3e73aca123e5-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.259158 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.259128 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pkcc\" (UniqueName: \"kubernetes.io/projected/e88d9911-2196-41cd-a9ae-3e73aca123e5-kube-api-access-7pkcc\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.259325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.259307 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-workload-certs\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.259373 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.259320 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-credential-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.259977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.259950 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88d9911-2196-41cd-a9ae-3e73aca123e5-istiod-ca-cert\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.259977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.259968 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-data\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.259977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.259955 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-workload-socket\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.261326 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.261304 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-envoy\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.261773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.261742 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-podinfo\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.283070 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.283044 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-token\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.283181 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.283103 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pkcc\" (UniqueName: \"kubernetes.io/projected/e88d9911-2196-41cd-a9ae-3e73aca123e5-kube-api-access-7pkcc\") pod \"e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.374865 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.374804 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:40.395010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.394977 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:40.552728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.552591 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:40.727962 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:25:40.727928 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88d9911_2196_41cd_a9ae_3e73aca123e5.slice/crio-c0bd2873151d5cfc3d6c04c17a4814a9d1fece7537862be35d7cc1c37235b1ef WatchSource:0}: Error finding container c0bd2873151d5cfc3d6c04c17a4814a9d1fece7537862be35d7cc1c37235b1ef: Status 404 returned error can't find the container with id c0bd2873151d5cfc3d6c04c17a4814a9d1fece7537862be35d7cc1c37235b1ef Oct 30 06:25:40.728645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.728618 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg"] Oct 30 06:25:40.730381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.730345 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:40.730479 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.730426 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:40.730479 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.730471 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:25:40.808707 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:40.808626 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:41.052519 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:41.052480 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" event={"ID":"e88d9911-2196-41cd-a9ae-3e73aca123e5","Type":"ContainerStarted","Data":"4843e594a62b341adfb645c2ee2ddaa62aec897dcd46339372629a748c652749"} Oct 30 06:25:41.052723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:41.052548 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" event={"ID":"e88d9911-2196-41cd-a9ae-3e73aca123e5","Type":"ContainerStarted","Data":"c0bd2873151d5cfc3d6c04c17a4814a9d1fece7537862be35d7cc1c37235b1ef"} Oct 30 06:25:41.099836 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:41.099718 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" podStartSLOduration=1.099700464 podStartE2EDuration="1.099700464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:41.097664016 +0000 UTC m=+1907.606683905" watchObservedRunningTime="2026-10-30 06:25:41.099700464 +0000 UTC m=+1907.608720351" Oct 30 06:25:41.375164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:41.375127 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:41.376790 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:41.376769 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:25:41.376905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:41.376813 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" podUID="e88d9911-2196-41cd-a9ae-3e73aca123e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 06:25:42.266935 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:42.266892 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:42.376019 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:42.375985 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:25:42.376232 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:42.376045 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" podUID="e88d9911-2196-41cd-a9ae-3e73aca123e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 06:25:43.375509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:43.375473 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:25:43.376118 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:43.376085 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" podUID="e88d9911-2196-41cd-a9ae-3e73aca123e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 06:25:44.208916 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:44.208882 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:44.375457 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:44.375427 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:25:44.375626 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:44.375478 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" podUID="e88d9911-2196-41cd-a9ae-3e73aca123e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 06:25:45.375862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:45.375827 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:25:45.376297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:45.375886 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" podUID="e88d9911-2196-41cd-a9ae-3e73aca123e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 06:25:45.708878 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:45.708790 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:45.875888 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:45.875858 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:46.221965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:46.221936 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:46.376159 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:46.376115 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:25:46.376573 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:46.376171 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" podUID="e88d9911-2196-41cd-a9ae-3e73aca123e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 06:25:47.375970 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.375932 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 06:25:47.376171 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.376001 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" podUID="e88d9911-2196-41cd-a9ae-3e73aca123e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 06:25:47.483041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.482995 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:47.647592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.647509 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4"] Oct 30 06:25:47.651440 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.651417 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" Oct 30 06:25:47.657980 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.657943 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:47.681399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.681371 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ec27ed\"" Oct 30 06:25:47.681609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.681375 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ec27ed-dockercfg-rmmll\"" Oct 30 06:25:47.684930 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.684910 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4"] Oct 30 06:25:47.687718 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.687696 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c"] Oct 30 06:25:47.691667 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.691648 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" Oct 30 06:25:47.710032 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.710006 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c"] Oct 30 06:25:47.779914 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.779365 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058369d8-43cd-4de2-98f6-5fac727ec3e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4\" (UID: \"058369d8-43cd-4de2-98f6-5fac727ec3e5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" Oct 30 06:25:47.779914 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.779627 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmh2n\" (UniqueName: \"kubernetes.io/projected/058369d8-43cd-4de2-98f6-5fac727ec3e5-kube-api-access-mmh2n\") pod \"payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4\" (UID: \"058369d8-43cd-4de2-98f6-5fac727ec3e5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" Oct 30 06:25:47.881148 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.881105 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmh2n\" (UniqueName: \"kubernetes.io/projected/058369d8-43cd-4de2-98f6-5fac727ec3e5-kube-api-access-mmh2n\") pod \"payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4\" (UID: \"058369d8-43cd-4de2-98f6-5fac727ec3e5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" Oct 30 06:25:47.881354 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.881155 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr7sm\" (UniqueName: \"kubernetes.io/projected/1e816f9e-3d95-4347-9b8c-71146fba6add-kube-api-access-mr7sm\") pod \"payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c\" (UID: \"1e816f9e-3d95-4347-9b8c-71146fba6add\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" Oct 30 06:25:47.881354 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.881287 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058369d8-43cd-4de2-98f6-5fac727ec3e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4\" (UID: \"058369d8-43cd-4de2-98f6-5fac727ec3e5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" Oct 30 06:25:47.881577 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.881557 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e816f9e-3d95-4347-9b8c-71146fba6add-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c\" (UID: \"1e816f9e-3d95-4347-9b8c-71146fba6add\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" Oct 30 06:25:47.882307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.882282 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058369d8-43cd-4de2-98f6-5fac727ec3e5-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4\" (UID: \"058369d8-43cd-4de2-98f6-5fac727ec3e5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" Oct 30 06:25:47.906342 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.906249 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmh2n\" (UniqueName: \"kubernetes.io/projected/058369d8-43cd-4de2-98f6-5fac727ec3e5-kube-api-access-mmh2n\") pod \"payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4\" (UID: \"058369d8-43cd-4de2-98f6-5fac727ec3e5\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" Oct 30 06:25:47.969556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.969528 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" Oct 30 06:25:47.982494 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.982466 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr7sm\" (UniqueName: \"kubernetes.io/projected/1e816f9e-3d95-4347-9b8c-71146fba6add-kube-api-access-mr7sm\") pod \"payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c\" (UID: \"1e816f9e-3d95-4347-9b8c-71146fba6add\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" Oct 30 06:25:47.982691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.982668 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e816f9e-3d95-4347-9b8c-71146fba6add-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c\" (UID: \"1e816f9e-3d95-4347-9b8c-71146fba6add\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" Oct 30 06:25:47.983480 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:47.983415 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e816f9e-3d95-4347-9b8c-71146fba6add-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c\" (UID: \"1e816f9e-3d95-4347-9b8c-71146fba6add\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" Oct 30 06:25:48.007009 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:48.006980 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr7sm\" (UniqueName: \"kubernetes.io/projected/1e816f9e-3d95-4347-9b8c-71146fba6add-kube-api-access-mr7sm\") pod \"payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c\" (UID: \"1e816f9e-3d95-4347-9b8c-71146fba6add\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" Oct 30 06:25:48.010751 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:48.010722 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" Oct 30 06:25:48.285703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:48.285679 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4"] Oct 30 06:25:48.288120 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:25:48.288086 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058369d8_43cd_4de2_98f6_5fac727ec3e5.slice/crio-d78f7d4b7f0146af9d2fc99d1e20e0b09fae20046535ed279980a4ae4c36d44f WatchSource:0}: Error finding container d78f7d4b7f0146af9d2fc99d1e20e0b09fae20046535ed279980a4ae4c36d44f: Status 404 returned error can't find the container with id d78f7d4b7f0146af9d2fc99d1e20e0b09fae20046535ed279980a4ae4c36d44f Oct 30 06:25:48.360873 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:48.360838 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c"] Oct 30 06:25:48.414927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:48.414901 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:48.415512 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:48.415488 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:48.417072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:48.417050 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:25:48.562994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:48.562905 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:49.114784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:49.114747 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" event={"ID":"058369d8-43cd-4de2-98f6-5fac727ec3e5","Type":"ContainerStarted","Data":"92544ff91c1bd65425d8b8cb3c21b2119598bb33dcc912889483bc769f8bc620"} Oct 30 06:25:49.115035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:49.114801 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" event={"ID":"058369d8-43cd-4de2-98f6-5fac727ec3e5","Type":"ContainerStarted","Data":"d78f7d4b7f0146af9d2fc99d1e20e0b09fae20046535ed279980a4ae4c36d44f"} Oct 30 06:25:49.115035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:49.114937 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" Oct 30 06:25:49.116393 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:49.116367 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" event={"ID":"1e816f9e-3d95-4347-9b8c-71146fba6add","Type":"ContainerStarted","Data":"d409548e1b67c8343ea899391be1dec004e0494261412f8a1875245610b2dae7"} Oct 30 06:25:49.116553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:49.116536 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" Oct 30 06:25:49.116613 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:49.116578 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" event={"ID":"1e816f9e-3d95-4347-9b8c-71146fba6add","Type":"ContainerStarted","Data":"c3f7ccdd82f513ed0f52580ccaccbed2463ead7d966022b70219b8f40468ba0f"} Oct 30 06:25:49.163491 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:49.163446 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" podStartSLOduration=2.163434265 podStartE2EDuration="2.163434265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:49.160306507 +0000 UTC m=+1915.669326394" watchObservedRunningTime="2026-10-30 06:25:49.163434265 +0000 UTC m=+1915.672454152" Oct 30 06:25:49.205004 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:49.204954 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" podStartSLOduration=2.204938166 podStartE2EDuration="2.204938166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:49.204590148 +0000 UTC m=+1915.713610047" watchObservedRunningTime="2026-10-30 06:25:49.204938166 +0000 UTC m=+1915.713958055" Oct 30 06:25:49.793950 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:49.793916 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:49.964783 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:49.964754 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:50.499082 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:50.499023 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:52.435519 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:52.435487 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:54.366426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.366389 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:54.411818 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.411781 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" podUID="52465882-affe-4707-8a97-10c1b5405857" containerName="main" containerID="cri-o://91bbac6c18fcb393cc8cf57cdd88fcfe6c0079fe4f0fc82c457486d4ed75bc7f" gracePeriod=45 Oct 30 06:25:54.706977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.706952 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:25:54.765335 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.765301 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-tmp-dir\") pod \"52465882-affe-4707-8a97-10c1b5405857\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " Oct 30 06:25:54.765528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.765378 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-model-cache\") pod \"52465882-affe-4707-8a97-10c1b5405857\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " Oct 30 06:25:54.765528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.765429 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-home\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-home\") pod \"52465882-affe-4707-8a97-10c1b5405857\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " Oct 30 06:25:54.765528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.765457 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52465882-affe-4707-8a97-10c1b5405857-kube-api-access-f4ng2\" (UniqueName: \"kubernetes.io/projected/52465882-affe-4707-8a97-10c1b5405857-kube-api-access-f4ng2\") pod \"52465882-affe-4707-8a97-10c1b5405857\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " Oct 30 06:25:54.765714 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.765530 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-dshm\") pod \"52465882-affe-4707-8a97-10c1b5405857\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " Oct 30 06:25:54.765714 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.765556 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-tmp-dir" pod "52465882-affe-4707-8a97-10c1b5405857" (UID: "52465882-affe-4707-8a97-10c1b5405857"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:54.765714 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.765651 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52465882-affe-4707-8a97-10c1b5405857-tls-certs\" (UniqueName: \"kubernetes.io/secret/52465882-affe-4707-8a97-10c1b5405857-tls-certs\") pod \"52465882-affe-4707-8a97-10c1b5405857\" (UID: \"52465882-affe-4707-8a97-10c1b5405857\") " Oct 30 06:25:54.765714 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.765657 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-model-cache" pod "52465882-affe-4707-8a97-10c1b5405857" (UID: "52465882-affe-4707-8a97-10c1b5405857"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:54.765929 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.765754 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-home" pod "52465882-affe-4707-8a97-10c1b5405857" (UID: "52465882-affe-4707-8a97-10c1b5405857"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:54.766251 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.766188 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:25:54.766456 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.766440 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:25:54.766577 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.766563 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:25:54.768609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.768579 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52465882-affe-4707-8a97-10c1b5405857-tls-certs" pod "52465882-affe-4707-8a97-10c1b5405857" (UID: "52465882-affe-4707-8a97-10c1b5405857"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:54.770039 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.770005 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52465882-affe-4707-8a97-10c1b5405857-kube-api-access-f4ng2" pod "52465882-affe-4707-8a97-10c1b5405857" (UID: "52465882-affe-4707-8a97-10c1b5405857"). InnerVolumeSpecName "kube-api-access-f4ng2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:54.770285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.770263 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-dshm" pod "52465882-affe-4707-8a97-10c1b5405857" (UID: "52465882-affe-4707-8a97-10c1b5405857"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:54.867759 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.867730 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52465882-affe-4707-8a97-10c1b5405857-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:25:54.867759 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.867759 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4ng2\" (UniqueName: \"kubernetes.io/projected/52465882-affe-4707-8a97-10c1b5405857-kube-api-access-f4ng2\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:25:54.867961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:54.867775 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52465882-affe-4707-8a97-10c1b5405857-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:25:55.153818 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:55.153780 6999 generic.go:356] "Generic (PLEG): container finished" podID="52465882-affe-4707-8a97-10c1b5405857" containerID="91bbac6c18fcb393cc8cf57cdd88fcfe6c0079fe4f0fc82c457486d4ed75bc7f" exitCode=0 Oct 30 06:25:55.153993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:55.153825 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" Oct 30 06:25:55.153993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:55.153858 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" event={"ID":"52465882-affe-4707-8a97-10c1b5405857","Type":"ContainerDied","Data":"91bbac6c18fcb393cc8cf57cdd88fcfe6c0079fe4f0fc82c457486d4ed75bc7f"} Oct 30 06:25:55.153993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:55.153915 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc" event={"ID":"52465882-affe-4707-8a97-10c1b5405857","Type":"ContainerDied","Data":"7d71a6d4feccc69b00a322ddcd924f93b775a15995125873f5979ea0072af221"} Oct 30 06:25:55.153993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:55.153938 6999 scope.go:122] "RemoveContainer" containerID="91bbac6c18fcb393cc8cf57cdd88fcfe6c0079fe4f0fc82c457486d4ed75bc7f" Oct 30 06:25:55.163609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:55.163586 6999 scope.go:122] "RemoveContainer" containerID="91bbac6c18fcb393cc8cf57cdd88fcfe6c0079fe4f0fc82c457486d4ed75bc7f" Oct 30 06:25:55.163888 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:25:55.163869 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91bbac6c18fcb393cc8cf57cdd88fcfe6c0079fe4f0fc82c457486d4ed75bc7f\": container with ID starting with 91bbac6c18fcb393cc8cf57cdd88fcfe6c0079fe4f0fc82c457486d4ed75bc7f not found: ID does not exist" containerID="91bbac6c18fcb393cc8cf57cdd88fcfe6c0079fe4f0fc82c457486d4ed75bc7f" Oct 30 06:25:55.163954 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:55.163900 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91bbac6c18fcb393cc8cf57cdd88fcfe6c0079fe4f0fc82c457486d4ed75bc7f"} err="failed to get container status \"91bbac6c18fcb393cc8cf57cdd88fcfe6c0079fe4f0fc82c457486d4ed75bc7f\": rpc error: code = NotFound desc = could not find container \"91bbac6c18fcb393cc8cf57cdd88fcfe6c0079fe4f0fc82c457486d4ed75bc7f\": container with ID starting with 91bbac6c18fcb393cc8cf57cdd88fcfe6c0079fe4f0fc82c457486d4ed75bc7f not found: ID does not exist" Oct 30 06:25:55.208948 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:55.208921 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc"] Oct 30 06:25:55.220405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:55.220384 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-dfa2ae/praxis-route-dfa2ae-kserve-6cc77c7b47-vg4rc"] Oct 30 06:25:55.608206 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:55.608160 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52465882-affe-4707-8a97-10c1b5405857" path="/var/lib/kubelet/pods/52465882-affe-4707-8a97-10c1b5405857/volumes" Oct 30 06:25:56.110698 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:56.110658 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:25:57.968228 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:25:57.968169 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:00.046441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:00.046402 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:00.124987 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:00.124961 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" Oct 30 06:26:00.125204 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:00.125131 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" Oct 30 06:26:00.874764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:00.874736 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:01.225691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:01.225611 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:01.569344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:01.569232 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:01.837066 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:01.836981 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87"] Oct 30 06:26:01.837673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:01.837652 6999 cpu_manager.go:414] "containerMap: removing container" podUID="52465882-affe-4707-8a97-10c1b5405857" containerName="main" Oct 30 06:26:01.837673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:01.837671 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52465882-affe-4707-8a97-10c1b5405857" containerName="main" Oct 30 06:26:01.837867 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:01.837767 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="52465882-affe-4707-8a97-10c1b5405857" containerName="main" Oct 30 06:26:01.842816 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:01.842793 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:01.874169 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:01.874144 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ec27ed-serving-cert\"" Oct 30 06:26:01.876786 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:01.876752 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87"] Oct 30 06:26:01.914471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:01.914430 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:01.970927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:01.970893 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c78c40-f1ad-46ea-bba5-807cdecde9c9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:01.971114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:01.970943 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c78c40-f1ad-46ea-bba5-807cdecde9c9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:01.971114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:01.971070 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx8lw\" (UniqueName: \"kubernetes.io/projected/26c78c40-f1ad-46ea-bba5-807cdecde9c9-kube-api-access-rx8lw\") pod \"maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:01.971114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:01.971097 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c78c40-f1ad-46ea-bba5-807cdecde9c9-metrics-certs\") pod \"maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:02.071891 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:02.071853 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c78c40-f1ad-46ea-bba5-807cdecde9c9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:02.072080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:02.071902 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c78c40-f1ad-46ea-bba5-807cdecde9c9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:02.072080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:02.071980 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx8lw\" (UniqueName: \"kubernetes.io/projected/26c78c40-f1ad-46ea-bba5-807cdecde9c9-kube-api-access-rx8lw\") pod \"maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:02.072199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:02.072102 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c78c40-f1ad-46ea-bba5-807cdecde9c9-metrics-certs\") pod \"maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:02.072746 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:02.072684 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c78c40-f1ad-46ea-bba5-807cdecde9c9-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:02.074972 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:02.074947 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c78c40-f1ad-46ea-bba5-807cdecde9c9-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:02.075260 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:02.075155 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c78c40-f1ad-46ea-bba5-807cdecde9c9-metrics-certs\") pod \"maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:02.092026 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:02.091917 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:02.096071 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:02.096048 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx8lw\" (UniqueName: \"kubernetes.io/projected/26c78c40-f1ad-46ea-bba5-807cdecde9c9-kube-api-access-rx8lw\") pod \"maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:02.158103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:02.158062 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:02.249054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:02.249015 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:02.476448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:02.476425 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87"] Oct 30 06:26:02.479094 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:26:02.479070 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c78c40_f1ad_46ea_bba5_807cdecde9c9.slice/crio-e310c9dd35918fd822d852efab53bc9cb3ef79f641325e1b5f6c2d1b3d43772d WatchSource:0}: Error finding container e310c9dd35918fd822d852efab53bc9cb3ef79f641325e1b5f6c2d1b3d43772d: Status 404 returned error can't find the container with id e310c9dd35918fd822d852efab53bc9cb3ef79f641325e1b5f6c2d1b3d43772d Oct 30 06:26:03.010680 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:03.010646 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:03.212327 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:03.212289 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" event={"ID":"26c78c40-f1ad-46ea-bba5-807cdecde9c9","Type":"ContainerStarted","Data":"ce356b235445479ac16833ba28f7cd2a6d5a5b9df22f25da5bd99607a411cdb6"} Oct 30 06:26:03.212537 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:03.212348 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" event={"ID":"26c78c40-f1ad-46ea-bba5-807cdecde9c9","Type":"ContainerStarted","Data":"e310c9dd35918fd822d852efab53bc9cb3ef79f641325e1b5f6c2d1b3d43772d"} Oct 30 06:26:03.212537 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:03.212432 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:03.256189 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:03.256138 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" podStartSLOduration=1.90093636 podStartE2EDuration="2.25612067s" totalImagesPullingTime="355.18431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:03.254400782 +0000 UTC m=+1929.763420669" watchObservedRunningTime="2026-10-30 06:26:03.25612067 +0000 UTC m=+1929.765140554" Oct 30 06:26:03.959955 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:03.959921 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:04.226923 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:26:04.211440 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:26:06.337294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:06.337258 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:06.533195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:06.533156 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:07.223378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.223341 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5"] Oct 30 06:26:07.226488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.226466 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.235956 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.235928 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t"] Oct 30 06:26:07.239568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.239550 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.256697 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.256667 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5"] Oct 30 06:26:07.276106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.276081 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t"] Oct 30 06:26:07.332707 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.332682 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.332887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.332721 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpjpl\" (UniqueName: \"kubernetes.io/projected/e17ee302-1c6d-4e12-af38-41ba781e7379-kube-api-access-gpjpl\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.332887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.332751 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.332887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.332842 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.333080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.332890 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.333080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.332907 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.333080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.332930 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqbsq\" (UniqueName: \"kubernetes.io/projected/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-kube-api-access-pqbsq\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.333080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.332946 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17ee302-1c6d-4e12-af38-41ba781e7379-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.333080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.332966 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.333080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.332984 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.333080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.333009 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.333080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.333023 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.333080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.333055 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.333080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.333076 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.333674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.333118 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.333674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.333134 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.333674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.333169 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.333674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.333187 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.336011 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.335986 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8d6349-openshift-default-dockercfg-xs2qm\"" Oct 30 06:26:07.434060 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434029 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.434569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434067 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.434569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434102 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.434569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434134 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.434569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434191 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.434569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434260 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.434569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434317 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.434569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434346 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.434569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434416 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.434569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434462 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpjpl\" (UniqueName: \"kubernetes.io/projected/e17ee302-1c6d-4e12-af38-41ba781e7379-kube-api-access-gpjpl\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.435046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434614 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.435046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434597 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.435046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434714 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.435046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434793 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.435046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434821 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.435046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434863 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqbsq\" (UniqueName: \"kubernetes.io/projected/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-kube-api-access-pqbsq\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.435046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434893 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17ee302-1c6d-4e12-af38-41ba781e7379-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.435046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434928 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.435046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434941 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.435046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434963 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.435046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.435037 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.435046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.434943 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.435821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.435425 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.435821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.435579 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.436030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.435989 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.436521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.436493 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17ee302-1c6d-4e12-af38-41ba781e7379-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.437170 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.437133 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.437309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.437186 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.437783 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.437740 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.437783 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.436515 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.438260 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.438201 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.438764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.438742 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.460844 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.460822 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqbsq\" (UniqueName: \"kubernetes.io/projected/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-kube-api-access-pqbsq\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.463458 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.463428 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.463633 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.463612 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpjpl\" (UniqueName: \"kubernetes.io/projected/e17ee302-1c6d-4e12-af38-41ba781e7379-kube-api-access-gpjpl\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.465880 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.465859 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.541788 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.541728 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:07.551952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.551920 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:07.880722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.880693 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5"] Oct 30 06:26:07.883203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.883172 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t"] Oct 30 06:26:07.885749 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:26:07.885721 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode17ee302_1c6d_4e12_af38_41ba781e7379.slice/crio-2d057154024c9aff9f2137c7615d5baaae142d6ac58a171dc069b68adbded14d WatchSource:0}: Error finding container 2d057154024c9aff9f2137c7615d5baaae142d6ac58a171dc069b68adbded14d: Status 404 returned error can't find the container with id 2d057154024c9aff9f2137c7615d5baaae142d6ac58a171dc069b68adbded14d Oct 30 06:26:07.888334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.888291 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:07.888410 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.888360 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:07.888410 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.888396 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:07.897286 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:26:07.897264 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3fb8b1_0bba_4b12_bc51_d9f137132e4d.slice/crio-af07b15bcd59ee816429f1612931810f67a9e2dd4390b6abbc58a5ee24d84abc WatchSource:0}: Error finding container af07b15bcd59ee816429f1612931810f67a9e2dd4390b6abbc58a5ee24d84abc: Status 404 returned error can't find the container with id af07b15bcd59ee816429f1612931810f67a9e2dd4390b6abbc58a5ee24d84abc Oct 30 06:26:07.899634 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.899602 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:07.899794 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.899677 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:07.899794 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:07.899725 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:08.105931 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:08.105884 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:08.255526 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:08.255440 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:08.256869 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:08.256587 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" event={"ID":"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d","Type":"ContainerStarted","Data":"d7ee6de12adec7c4f813f2a858fa0e72c7c825560be9be442ca77f40223e0294"} Oct 30 06:26:08.256869 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:08.256673 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" event={"ID":"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d","Type":"ContainerStarted","Data":"af07b15bcd59ee816429f1612931810f67a9e2dd4390b6abbc58a5ee24d84abc"} Oct 30 06:26:08.258942 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:08.258919 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" event={"ID":"e17ee302-1c6d-4e12-af38-41ba781e7379","Type":"ContainerStarted","Data":"d579a78c7b5ab0f1e409ce9324ca76e318284ed465cbae252f1c21b5598e92c9"} Oct 30 06:26:08.259057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:08.258963 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" event={"ID":"e17ee302-1c6d-4e12-af38-41ba781e7379","Type":"ContainerStarted","Data":"2d057154024c9aff9f2137c7615d5baaae142d6ac58a171dc069b68adbded14d"} Oct 30 06:26:08.308240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:08.305177 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" podStartSLOduration=1.305159784 podStartE2EDuration="1.305159784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:08.304358394 +0000 UTC m=+1934.813378306" watchObservedRunningTime="2026-10-30 06:26:08.305159784 +0000 UTC m=+1934.814179672" Oct 30 06:26:08.398282 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:08.394005 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" podStartSLOduration=1.393982441 podStartE2EDuration="1.393982441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:08.356829601 +0000 UTC m=+1934.865849498" watchObservedRunningTime="2026-10-30 06:26:08.393982441 +0000 UTC m=+1934.903002329" Oct 30 06:26:08.541977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:08.541896 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:08.543247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:08.543187 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 06:26:08.543395 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:08.543282 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" podUID="e17ee302-1c6d-4e12-af38-41ba781e7379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 06:26:08.552784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:08.552763 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:08.554033 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:08.554014 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 06:26:08.554105 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:08.554063 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" podUID="0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 06:26:08.656488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:08.656455 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:09.229799 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:09.229777 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:26:09.548297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:09.543713 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 06:26:09.548297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:09.543811 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" podUID="e17ee302-1c6d-4e12-af38-41ba781e7379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 06:26:09.563241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:09.552587 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 06:26:09.563241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:09.552643 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" podUID="0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 06:26:10.550239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:10.542607 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 06:26:10.550239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:10.542669 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" podUID="e17ee302-1c6d-4e12-af38-41ba781e7379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 06:26:10.552601 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:10.552571 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 06:26:10.552830 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:10.552804 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" podUID="0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 06:26:10.808421 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:10.808334 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:11.545666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:11.545627 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 06:26:11.545980 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:11.545947 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" podUID="e17ee302-1c6d-4e12-af38-41ba781e7379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 06:26:11.567237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:11.556623 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 06:26:11.567237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:11.556677 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" podUID="0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 06:26:12.550685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:12.550650 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 06:26:12.550997 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:12.550967 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" podUID="e17ee302-1c6d-4e12-af38-41ba781e7379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 06:26:12.555320 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:12.555292 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 06:26:12.555507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:12.555486 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" podUID="0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 06:26:12.898857 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:12.898821 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:13.559255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:13.558610 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 06:26:13.559255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:13.558668 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" podUID="e17ee302-1c6d-4e12-af38-41ba781e7379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 06:26:13.563234 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:13.559885 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 06:26:13.563234 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:13.560091 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" podUID="0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 06:26:14.542168 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:14.542128 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 06:26:14.542664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:14.542198 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" podUID="e17ee302-1c6d-4e12-af38-41ba781e7379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 06:26:14.552891 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:14.552864 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 06:26:14.553062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:14.552912 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" podUID="0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 06:26:14.839257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:14.839086 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:15.729086 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:15.728991 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:15.729861 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:15.729823 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:15.731269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:15.731238 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:26:15.735112 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:15.735092 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:15.735533 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:15.735485 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:15.737333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:15.737310 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:26:16.758194 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:16.758138 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:17.435648 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:17.435606 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:17.983912 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:17.983878 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:18.666851 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:18.666822 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:20.554569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:20.554529 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:22.382567 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:22.382537 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:24.071868 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.071829 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g"] Oct 30 06:26:24.076050 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.076026 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:24.096351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.096167 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-adcdcb-serving-cert\"" Oct 30 06:26:24.109479 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.109451 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g"] Oct 30 06:26:24.127893 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.127871 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:24.128007 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.127984 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:24.128055 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.128009 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dceee649-894a-4330-a9df-0097ca9e352e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:24.128089 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.128067 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c544z\" (UniqueName: \"kubernetes.io/projected/dceee649-894a-4330-a9df-0097ca9e352e-kube-api-access-c544z\") pod \"maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:24.204736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.204709 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:24.229080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.229045 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:24.229239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.229087 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dceee649-894a-4330-a9df-0097ca9e352e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:24.229239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.229127 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c544z\" (UniqueName: \"kubernetes.io/projected/dceee649-894a-4330-a9df-0097ca9e352e-kube-api-access-c544z\") pod \"maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:24.229381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.229326 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:24.229447 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:26:24.229415 6999 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-serving-cert: secret "maas-api-e2e-shared-a-w1-adcdcb-serving-cert" not found Oct 30 06:26:24.229505 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:26:24.229493 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-maas-api-tls podName:dceee649-894a-4330-a9df-0097ca9e352e nodeName:}" failed. No retries permitted until 2026-10-30 06:26:24.729472044 +0000 UTC m=+1951.238491915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-maas-api-tls") pod "maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" (UID: "dceee649-894a-4330-a9df-0097ca9e352e") : secret "maas-api-e2e-shared-a-w1-adcdcb-serving-cert" not found Oct 30 06:26:24.229842 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.229820 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dceee649-894a-4330-a9df-0097ca9e352e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:24.231718 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.231699 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:24.253297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.253276 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c544z\" (UniqueName: \"kubernetes.io/projected/dceee649-894a-4330-a9df-0097ca9e352e-kube-api-access-c544z\") pod \"maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:24.352347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.352268 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:24.503746 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.503713 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:24.734613 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.734581 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:24.737319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.737295 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:24.990296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:24.990192 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:25.155732 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:25.155583 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:25.301120 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:25.301043 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:25.314985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:25.314953 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g"] Oct 30 06:26:25.316552 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:26:25.316525 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddceee649_894a_4330_a9df_0097ca9e352e.slice/crio-4f376b7f34ab57774afc6ca7f95ba37314a53f38d90acdf9637c031fea8505da WatchSource:0}: Error finding container 4f376b7f34ab57774afc6ca7f95ba37314a53f38d90acdf9637c031fea8505da: Status 404 returned error can't find the container with id 4f376b7f34ab57774afc6ca7f95ba37314a53f38d90acdf9637c031fea8505da Oct 30 06:26:25.413288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:25.413242 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" event={"ID":"dceee649-894a-4330-a9df-0097ca9e352e","Type":"ContainerStarted","Data":"4f376b7f34ab57774afc6ca7f95ba37314a53f38d90acdf9637c031fea8505da"} Oct 30 06:26:25.476754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:25.476718 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:26.324073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:26.324037 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:26.438724 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:26.438689 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" event={"ID":"dceee649-894a-4330-a9df-0097ca9e352e","Type":"ContainerStarted","Data":"d9b87b5c22c75cb1610b292e84c590d4b9fcbff701acce8a765c007ba69b3662"} Oct 30 06:26:26.438936 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:26.438759 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:26.484411 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:26.484379 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:26.486107 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:26.486061 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" podStartSLOduration=2.115022902 podStartE2EDuration="2.486048275s" totalImagesPullingTime="371.025373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:26.482801383 +0000 UTC m=+1952.991821269" watchObservedRunningTime="2026-10-30 06:26:26.486048275 +0000 UTC m=+1952.995068162" Oct 30 06:26:27.130034 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:27.129997 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:27.346404 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:27.346372 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:28.575528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:28.575498 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:30.723451 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:30.723423 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:31.060070 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:31.059987 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:31.210841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:31.210810 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:32.449353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:32.449327 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:26:32.781152 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:32.781068 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:33.151103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:33.151054 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:34.142552 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:26:34.142513 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:26:34.985263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:34.985227 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:35.832803 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:35.832755 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:36.327350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:36.327312 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:36.491289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:36.491254 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:36.897022 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:36.896982 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:37.084806 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:37.084766 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:37.237206 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:37.237127 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:38.794641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:38.794611 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:40.254955 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:40.254919 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:40.444169 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:40.444140 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:40.600775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:40.600655 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:41.007166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:41.007126 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:42.228771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.228731 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:42.439544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.439514 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:42.608018 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.607934 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll"] Oct 30 06:26:42.613773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.613752 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" Oct 30 06:26:42.617740 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.617713 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:42.632710 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.632691 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll"] Oct 30 06:26:42.690235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.689454 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-6e2548\"" Oct 30 06:26:42.690235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.689594 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-6e2548-dockercfg-xt2bl\"" Oct 30 06:26:42.738611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.711316 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpvg4\" (UniqueName: \"kubernetes.io/projected/7bc29aa6-9b21-4581-94f3-3e5302cc4b6b-kube-api-access-mpvg4\") pod \"payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll\" (UID: \"7bc29aa6-9b21-4581-94f3-3e5302cc4b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" Oct 30 06:26:42.738611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.711415 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc29aa6-9b21-4581-94f3-3e5302cc4b6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll\" (UID: \"7bc29aa6-9b21-4581-94f3-3e5302cc4b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" Oct 30 06:26:42.761234 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.759284 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb"] Oct 30 06:26:42.761234 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.759510 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" podUID="97503283-aa66-4f51-b666-c4eff11abc62" containerName="maas-api" containerID="cri-o://1a0156b950b7de0cd5de0c53f0d41a63b74e9c761d9339758d23d716f2b12332" gracePeriod=30 Oct 30 06:26:42.809049 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.809016 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:42.812517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.812486 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc29aa6-9b21-4581-94f3-3e5302cc4b6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll\" (UID: \"7bc29aa6-9b21-4581-94f3-3e5302cc4b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" Oct 30 06:26:42.812740 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.812721 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpvg4\" (UniqueName: \"kubernetes.io/projected/7bc29aa6-9b21-4581-94f3-3e5302cc4b6b-kube-api-access-mpvg4\") pod \"payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll\" (UID: \"7bc29aa6-9b21-4581-94f3-3e5302cc4b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" Oct 30 06:26:42.813376 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.813351 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc29aa6-9b21-4581-94f3-3e5302cc4b6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll\" (UID: \"7bc29aa6-9b21-4581-94f3-3e5302cc4b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" Oct 30 06:26:42.836634 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.836610 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpvg4\" (UniqueName: \"kubernetes.io/projected/7bc29aa6-9b21-4581-94f3-3e5302cc4b6b-kube-api-access-mpvg4\") pod \"payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll\" (UID: \"7bc29aa6-9b21-4581-94f3-3e5302cc4b6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" Oct 30 06:26:42.928621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:42.928588 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" Oct 30 06:26:43.153500 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.152405 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:26:43.219190 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.219136 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97503283-aa66-4f51-b666-c4eff11abc62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97503283-aa66-4f51-b666-c4eff11abc62-openshift-service-ca\") pod \"97503283-aa66-4f51-b666-c4eff11abc62\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " Oct 30 06:26:43.220065 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.220035 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97503283-aa66-4f51-b666-c4eff11abc62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/97503283-aa66-4f51-b666-c4eff11abc62-metrics-certs\") pod \"97503283-aa66-4f51-b666-c4eff11abc62\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " Oct 30 06:26:43.220546 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.220205 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97503283-aa66-4f51-b666-c4eff11abc62-kube-api-access-dgmhx\" (UniqueName: \"kubernetes.io/projected/97503283-aa66-4f51-b666-c4eff11abc62-kube-api-access-dgmhx\") pod \"97503283-aa66-4f51-b666-c4eff11abc62\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " Oct 30 06:26:43.220675 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.220625 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97503283-aa66-4f51-b666-c4eff11abc62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97503283-aa66-4f51-b666-c4eff11abc62-maas-api-tls\") pod \"97503283-aa66-4f51-b666-c4eff11abc62\" (UID: \"97503283-aa66-4f51-b666-c4eff11abc62\") " Oct 30 06:26:43.222552 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.222525 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97503283-aa66-4f51-b666-c4eff11abc62-openshift-service-ca" pod "97503283-aa66-4f51-b666-c4eff11abc62" (UID: "97503283-aa66-4f51-b666-c4eff11abc62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:43.225656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.225621 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97503283-aa66-4f51-b666-c4eff11abc62-kube-api-access-dgmhx" pod "97503283-aa66-4f51-b666-c4eff11abc62" (UID: "97503283-aa66-4f51-b666-c4eff11abc62"). InnerVolumeSpecName "kube-api-access-dgmhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:43.236587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.236555 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97503283-aa66-4f51-b666-c4eff11abc62-metrics-certs" pod "97503283-aa66-4f51-b666-c4eff11abc62" (UID: "97503283-aa66-4f51-b666-c4eff11abc62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:43.265551 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.265505 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97503283-aa66-4f51-b666-c4eff11abc62-maas-api-tls" pod "97503283-aa66-4f51-b666-c4eff11abc62" (UID: "97503283-aa66-4f51-b666-c4eff11abc62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:43.310508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.310387 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll"] Oct 30 06:26:43.312553 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:26:43.312524 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc29aa6_9b21_4581_94f3_3e5302cc4b6b.slice/crio-00d61b120212d77bc6ad25a0eb0cab97e54cd53cd2adbedd3ae6ddd1a85cd7a9 WatchSource:0}: Error finding container 00d61b120212d77bc6ad25a0eb0cab97e54cd53cd2adbedd3ae6ddd1a85cd7a9: Status 404 returned error can't find the container with id 00d61b120212d77bc6ad25a0eb0cab97e54cd53cd2adbedd3ae6ddd1a85cd7a9 Oct 30 06:26:43.322884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.322859 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgmhx\" (UniqueName: \"kubernetes.io/projected/97503283-aa66-4f51-b666-c4eff11abc62-kube-api-access-dgmhx\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:26:43.322994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.322890 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/97503283-aa66-4f51-b666-c4eff11abc62-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:26:43.322994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.322904 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/97503283-aa66-4f51-b666-c4eff11abc62-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:26:43.322994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.322922 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97503283-aa66-4f51-b666-c4eff11abc62-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:26:43.490335 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.490262 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:43.622432 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.622397 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" event={"ID":"7bc29aa6-9b21-4581-94f3-3e5302cc4b6b","Type":"ContainerStarted","Data":"2ad8b9d3c136fc93ec2f6a9625f88246b841ae5e5e51a134c4f88cbb3aef1545"} Oct 30 06:26:43.622591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.622455 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" event={"ID":"7bc29aa6-9b21-4581-94f3-3e5302cc4b6b","Type":"ContainerStarted","Data":"00d61b120212d77bc6ad25a0eb0cab97e54cd53cd2adbedd3ae6ddd1a85cd7a9"} Oct 30 06:26:43.622591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.622488 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" Oct 30 06:26:43.623836 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.623808 6999 generic.go:356] "Generic (PLEG): container finished" podID="97503283-aa66-4f51-b666-c4eff11abc62" containerID="1a0156b950b7de0cd5de0c53f0d41a63b74e9c761d9339758d23d716f2b12332" exitCode=0 Oct 30 06:26:43.623954 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.623863 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" Oct 30 06:26:43.623954 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.623882 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" event={"ID":"97503283-aa66-4f51-b666-c4eff11abc62","Type":"ContainerDied","Data":"1a0156b950b7de0cd5de0c53f0d41a63b74e9c761d9339758d23d716f2b12332"} Oct 30 06:26:43.623954 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.623922 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb" event={"ID":"97503283-aa66-4f51-b666-c4eff11abc62","Type":"ContainerDied","Data":"8f13dfc4a5c54c49230bfe07fd54ece90dc387dddf1235693538b8042be167c3"} Oct 30 06:26:43.623954 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.623944 6999 scope.go:122] "RemoveContainer" containerID="1a0156b950b7de0cd5de0c53f0d41a63b74e9c761d9339758d23d716f2b12332" Oct 30 06:26:43.633162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.633145 6999 scope.go:122] "RemoveContainer" containerID="1a0156b950b7de0cd5de0c53f0d41a63b74e9c761d9339758d23d716f2b12332" Oct 30 06:26:43.633426 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:26:43.633407 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a0156b950b7de0cd5de0c53f0d41a63b74e9c761d9339758d23d716f2b12332\": container with ID starting with 1a0156b950b7de0cd5de0c53f0d41a63b74e9c761d9339758d23d716f2b12332 not found: ID does not exist" containerID="1a0156b950b7de0cd5de0c53f0d41a63b74e9c761d9339758d23d716f2b12332" Oct 30 06:26:43.633498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.633436 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a0156b950b7de0cd5de0c53f0d41a63b74e9c761d9339758d23d716f2b12332"} err="failed to get container status \"1a0156b950b7de0cd5de0c53f0d41a63b74e9c761d9339758d23d716f2b12332\": rpc error: code = NotFound desc = could not find container \"1a0156b950b7de0cd5de0c53f0d41a63b74e9c761d9339758d23d716f2b12332\": container with ID starting with 1a0156b950b7de0cd5de0c53f0d41a63b74e9c761d9339758d23d716f2b12332 not found: ID does not exist" Oct 30 06:26:43.656836 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.656809 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:43.672347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.672289 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" podStartSLOduration=1.672275925 podStartE2EDuration="1.672275925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:43.669915988 +0000 UTC m=+1970.178935867" watchObservedRunningTime="2026-10-30 06:26:43.672275925 +0000 UTC m=+1970.181295850" Oct 30 06:26:43.729237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.714165 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb"] Oct 30 06:26:43.729237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.714274 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-5a6371-69776db547-zvgfb"] Oct 30 06:26:43.812574 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:43.812500 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:44.201149 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:44.201118 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:45.608889 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:45.608855 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97503283-aa66-4f51-b666-c4eff11abc62" path="/var/lib/kubelet/pods/97503283-aa66-4f51-b666-c4eff11abc62/volumes" Oct 30 06:26:45.660586 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:45.660558 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:47.033431 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.033377 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:47.373128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.373098 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n"] Oct 30 06:26:47.373353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.373325 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" podUID="0191799b-7939-49e1-b3b9-496270844a86" containerName="maas-api" containerID="cri-o://a9411f19e64162a2a9fa86a414c584f5883c99e16d7e069ed235f601d2c77bb6" gracePeriod=30 Oct 30 06:26:47.572698 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.570132 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:47.653233 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.653138 6999 generic.go:356] "Generic (PLEG): container finished" podID="0191799b-7939-49e1-b3b9-496270844a86" containerID="a9411f19e64162a2a9fa86a414c584f5883c99e16d7e069ed235f601d2c77bb6" exitCode=0 Oct 30 06:26:47.653367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.653346 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" event={"ID":"0191799b-7939-49e1-b3b9-496270844a86","Type":"ContainerDied","Data":"a9411f19e64162a2a9fa86a414c584f5883c99e16d7e069ed235f601d2c77bb6"} Oct 30 06:26:47.688808 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.688780 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:26:47.733255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.733222 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:47.777252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.774725 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0191799b-7939-49e1-b3b9-496270844a86-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0191799b-7939-49e1-b3b9-496270844a86-maas-api-tls\") pod \"0191799b-7939-49e1-b3b9-496270844a86\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " Oct 30 06:26:47.777252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.774814 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0191799b-7939-49e1-b3b9-496270844a86-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0191799b-7939-49e1-b3b9-496270844a86-metrics-certs\") pod \"0191799b-7939-49e1-b3b9-496270844a86\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " Oct 30 06:26:47.777252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.774901 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0191799b-7939-49e1-b3b9-496270844a86-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0191799b-7939-49e1-b3b9-496270844a86-openshift-service-ca\") pod \"0191799b-7939-49e1-b3b9-496270844a86\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " Oct 30 06:26:47.777252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.774930 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0191799b-7939-49e1-b3b9-496270844a86-kube-api-access-dm7p6\" (UniqueName: \"kubernetes.io/projected/0191799b-7939-49e1-b3b9-496270844a86-kube-api-access-dm7p6\") pod \"0191799b-7939-49e1-b3b9-496270844a86\" (UID: \"0191799b-7939-49e1-b3b9-496270844a86\") " Oct 30 06:26:47.777252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.776941 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0191799b-7939-49e1-b3b9-496270844a86-openshift-service-ca" pod "0191799b-7939-49e1-b3b9-496270844a86" (UID: "0191799b-7939-49e1-b3b9-496270844a86"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:47.800164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.779863 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0191799b-7939-49e1-b3b9-496270844a86-maas-api-tls" pod "0191799b-7939-49e1-b3b9-496270844a86" (UID: "0191799b-7939-49e1-b3b9-496270844a86"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:47.800164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.797898 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0191799b-7939-49e1-b3b9-496270844a86-metrics-certs" pod "0191799b-7939-49e1-b3b9-496270844a86" (UID: "0191799b-7939-49e1-b3b9-496270844a86"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:47.804724 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.804691 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0191799b-7939-49e1-b3b9-496270844a86-kube-api-access-dm7p6" pod "0191799b-7939-49e1-b3b9-496270844a86" (UID: "0191799b-7939-49e1-b3b9-496270844a86"). InnerVolumeSpecName "kube-api-access-dm7p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:47.876854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.876821 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0191799b-7939-49e1-b3b9-496270844a86-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:26:47.876854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.876852 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0191799b-7939-49e1-b3b9-496270844a86-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:26:47.876854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.876862 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0191799b-7939-49e1-b3b9-496270844a86-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:26:47.877072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:47.876871 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm7p6\" (UniqueName: \"kubernetes.io/projected/0191799b-7939-49e1-b3b9-496270844a86-kube-api-access-dm7p6\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:26:48.080588 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.080515 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:48.284580 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.284548 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t"] Oct 30 06:26:48.285074 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.285062 6999 cpu_manager.go:414] "containerMap: removing container" podUID="0191799b-7939-49e1-b3b9-496270844a86" containerName="maas-api" Oct 30 06:26:48.285116 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.285075 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0191799b-7939-49e1-b3b9-496270844a86" containerName="maas-api" Oct 30 06:26:48.285116 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.285101 6999 cpu_manager.go:414] "containerMap: removing container" podUID="97503283-aa66-4f51-b666-c4eff11abc62" containerName="maas-api" Oct 30 06:26:48.285116 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.285106 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97503283-aa66-4f51-b666-c4eff11abc62" containerName="maas-api" Oct 30 06:26:48.285250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.285207 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="0191799b-7939-49e1-b3b9-496270844a86" containerName="maas-api" Oct 30 06:26:48.285297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.285255 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="97503283-aa66-4f51-b666-c4eff11abc62" containerName="maas-api" Oct 30 06:26:48.288623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.288607 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:48.337552 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.337454 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-645f2b-serving-cert\"" Oct 30 06:26:48.338876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.338846 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t"] Oct 30 06:26:48.378980 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.378941 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:48.395097 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.395067 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:48.395354 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.395118 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:48.395354 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.395324 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:48.395952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.395383 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rjpf\" (UniqueName: \"kubernetes.io/projected/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-kube-api-access-4rjpf\") pod \"maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:48.496490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.496451 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:48.496658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.496509 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:48.496658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.496600 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:48.496777 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.496655 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rjpf\" (UniqueName: \"kubernetes.io/projected/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-kube-api-access-4rjpf\") pod \"maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:48.497584 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.497553 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:48.499621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.499202 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:48.499621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.499495 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:48.521872 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.521833 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rjpf\" (UniqueName: \"kubernetes.io/projected/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-kube-api-access-4rjpf\") pod \"maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:48.546249 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.546177 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:48.605181 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.605110 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:48.670371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.667756 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" event={"ID":"0191799b-7939-49e1-b3b9-496270844a86","Type":"ContainerDied","Data":"1ac56a160d4f7914779cdb1f6961ca094c55bf3b34614410ff3152fcc8f2a20f"} Oct 30 06:26:48.670371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.667812 6999 scope.go:122] "RemoveContainer" containerID="a9411f19e64162a2a9fa86a414c584f5883c99e16d7e069ed235f601d2c77bb6" Oct 30 06:26:48.670371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.667922 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n" Oct 30 06:26:48.725370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.725310 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n"] Oct 30 06:26:48.730312 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.730284 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f5e55f-6fb888bc45-r7d9n"] Oct 30 06:26:48.913883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:48.913853 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t"] Oct 30 06:26:48.916418 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:26:48.916393 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6752f6_ee5f_4800_8fd8_b9df6aa69274.slice/crio-dd8663bba8887d332c100e8abf772eff3d339c3c5e497c023dfb82f63b1d8d97 WatchSource:0}: Error finding container dd8663bba8887d332c100e8abf772eff3d339c3c5e497c023dfb82f63b1d8d97: Status 404 returned error can't find the container with id dd8663bba8887d332c100e8abf772eff3d339c3c5e497c023dfb82f63b1d8d97 Oct 30 06:26:49.024924 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:49.024889 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:49.610555 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:49.610523 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0191799b-7939-49e1-b3b9-496270844a86" path="/var/lib/kubelet/pods/0191799b-7939-49e1-b3b9-496270844a86/volumes" Oct 30 06:26:49.677128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:49.677100 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" event={"ID":"3e6752f6-ee5f-4800-8fd8-b9df6aa69274","Type":"ContainerStarted","Data":"391e065825910030537fb1981c4a0cfab155a3f5246fb66edb1bb3f4ccfa721f"} Oct 30 06:26:49.677325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:49.677144 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" event={"ID":"3e6752f6-ee5f-4800-8fd8-b9df6aa69274","Type":"ContainerStarted","Data":"dd8663bba8887d332c100e8abf772eff3d339c3c5e497c023dfb82f63b1d8d97"} Oct 30 06:26:49.677405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:49.677338 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:49.721802 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:49.721756 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" podStartSLOduration=1.393148097 podStartE2EDuration="1.721743014s" totalImagesPullingTime="328.594917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:49.719327222 +0000 UTC m=+1976.228347110" watchObservedRunningTime="2026-10-30 06:26:49.721743014 +0000 UTC m=+1976.230762905" Oct 30 06:26:50.570365 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:50.570328 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:51.331048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:51.331012 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:51.473850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:51.473823 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:51.700859 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:51.700826 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:52.370635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.370599 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:52.380726 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.380701 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh"] Oct 30 06:26:52.385171 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.385150 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.413969 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.413944 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh"] Oct 30 06:26:52.523565 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.523536 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:52.549291 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.549264 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.549455 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.549312 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.549500 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.549466 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.549628 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.549606 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.549701 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.549648 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.549774 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.549757 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.549822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.549801 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5nf5\" (UniqueName: \"kubernetes.io/projected/a106c2d8-c5a5-4e22-b75d-922be5d0231a-kube-api-access-p5nf5\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.549859 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.549832 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.549920 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.549862 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.650956 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.650871 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.650956 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.650910 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.650956 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.650955 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.651259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.650974 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5nf5\" (UniqueName: \"kubernetes.io/projected/a106c2d8-c5a5-4e22-b75d-922be5d0231a-kube-api-access-p5nf5\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.651259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.651016 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.651259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.651050 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.651259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.651106 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.651259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.651146 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.651539 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.651283 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.651539 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.651520 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-credential-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.652422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.652395 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-workload-socket\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.652654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.652468 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-data\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.652654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.652534 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-workload-certs\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.652814 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.652790 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istiod-ca-cert\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.653771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.653743 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-envoy\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.654048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.654029 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-podinfo\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.700241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.692806 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-token\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.700241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.693325 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:52.704235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.701055 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5nf5\" (UniqueName: \"kubernetes.io/projected/a106c2d8-c5a5-4e22-b75d-922be5d0231a-kube-api-access-p5nf5\") pod \"e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.714552 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.708332 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:52.854247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:52.850074 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:53.085585 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:53.085496 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh"] Oct 30 06:26:53.089974 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:26:53.089948 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda106c2d8_c5a5_4e22_b75d_922be5d0231a.slice/crio-b579b3d26a1abaddee98844c678a3acc8a6d2deb1378a1befc2ba6a22375f648 WatchSource:0}: Error finding container b579b3d26a1abaddee98844c678a3acc8a6d2deb1378a1befc2ba6a22375f648: Status 404 returned error can't find the container with id b579b3d26a1abaddee98844c678a3acc8a6d2deb1378a1befc2ba6a22375f648 Oct 30 06:26:53.092666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:53.092628 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:53.092784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:53.092711 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:53.092784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:53.092757 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:26:53.219808 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:53.219770 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:53.406237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:53.405693 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:53.760293 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:53.759434 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" event={"ID":"a106c2d8-c5a5-4e22-b75d-922be5d0231a","Type":"ContainerStarted","Data":"f6f5f2b72cf4408bf0832cbdbc7e82b60f2e62b71c70fd721466758fb090a29b"} Oct 30 06:26:53.760293 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:53.759499 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" event={"ID":"a106c2d8-c5a5-4e22-b75d-922be5d0231a","Type":"ContainerStarted","Data":"b579b3d26a1abaddee98844c678a3acc8a6d2deb1378a1befc2ba6a22375f648"} Oct 30 06:26:53.819242 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:53.819080 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" podStartSLOduration=1.819063169 podStartE2EDuration="1.819063169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:53.818297204 +0000 UTC m=+1980.327317091" watchObservedRunningTime="2026-10-30 06:26:53.819063169 +0000 UTC m=+1980.328083056" Oct 30 06:26:54.202049 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.202015 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:54.337561 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.337527 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq"] Oct 30 06:26:54.341607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.341583 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" Oct 30 06:26:54.386130 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.385345 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-fa5a69\"" Oct 30 06:26:54.386130 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.385436 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-fa5a69-dockercfg-df6h7\"" Oct 30 06:26:54.390321 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.389533 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq"] Oct 30 06:26:54.406248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.405508 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpdxb\" (UniqueName: \"kubernetes.io/projected/f977329f-2b08-42f3-9359-154994886be9-kube-api-access-zpdxb\") pod \"payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq\" (UID: \"f977329f-2b08-42f3-9359-154994886be9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" Oct 30 06:26:54.406248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.405609 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f977329f-2b08-42f3-9359-154994886be9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq\" (UID: \"f977329f-2b08-42f3-9359-154994886be9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" Oct 30 06:26:54.417007 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.416895 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:54.520021 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.519903 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpdxb\" (UniqueName: \"kubernetes.io/projected/f977329f-2b08-42f3-9359-154994886be9-kube-api-access-zpdxb\") pod \"payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq\" (UID: \"f977329f-2b08-42f3-9359-154994886be9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" Oct 30 06:26:54.604693 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.604657 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f977329f-2b08-42f3-9359-154994886be9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq\" (UID: \"f977329f-2b08-42f3-9359-154994886be9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" Oct 30 06:26:54.633272 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.610634 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpdxb\" (UniqueName: \"kubernetes.io/projected/f977329f-2b08-42f3-9359-154994886be9-kube-api-access-zpdxb\") pod \"payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq\" (UID: \"f977329f-2b08-42f3-9359-154994886be9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" Oct 30 06:26:54.633272 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.611470 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f977329f-2b08-42f3-9359-154994886be9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq\" (UID: \"f977329f-2b08-42f3-9359-154994886be9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" Oct 30 06:26:54.667141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.667107 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" Oct 30 06:26:54.684264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.683194 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" Oct 30 06:26:54.742808 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.727860 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:26:54.806607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.798366 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 06:26:54.806607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:54.798440 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" podUID="a106c2d8-c5a5-4e22-b75d-922be5d0231a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 06:26:55.255108 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:55.255073 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:55.725239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:55.716255 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 06:26:55.725239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:55.716336 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" podUID="a106c2d8-c5a5-4e22-b75d-922be5d0231a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 06:26:55.761911 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:55.761881 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:26:55.816278 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:55.815822 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq"] Oct 30 06:26:56.010148 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:56.010119 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" event={"ID":"f977329f-2b08-42f3-9359-154994886be9","Type":"ContainerStarted","Data":"5108ab9515ce6666c640e1353f61ea9cecfb475cb93007cfbd398a5271d18b17"} Oct 30 06:26:56.396966 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:56.396938 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:56.560241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:56.555701 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:56.709615 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:56.709533 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 06:26:56.709992 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:56.709831 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" podUID="a106c2d8-c5a5-4e22-b75d-922be5d0231a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 06:26:56.807028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:56.806978 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:57.017598 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:57.017504 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" event={"ID":"f977329f-2b08-42f3-9359-154994886be9","Type":"ContainerStarted","Data":"a9f73eeadaa81e4433aa93402a85a2648d6f9e3fbb882eeb7b22cbfcb8ad85ab"} Oct 30 06:26:57.017752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:57.017625 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" Oct 30 06:26:57.066239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:57.065983 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" podStartSLOduration=3.065969487 podStartE2EDuration="3.065969487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:57.065624968 +0000 UTC m=+1983.574644867" watchObservedRunningTime="2026-10-30 06:26:57.065969487 +0000 UTC m=+1983.574989372" Oct 30 06:26:57.198261 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:57.198233 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:57.367830 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:57.367800 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:57.530734 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:57.530697 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:57.709589 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:57.709507 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 06:26:57.709589 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:57.709564 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" podUID="a106c2d8-c5a5-4e22-b75d-922be5d0231a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 06:26:58.709067 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:58.709032 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 06:26:58.709597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:58.709116 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" podUID="a106c2d8-c5a5-4e22-b75d-922be5d0231a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 06:26:58.972941 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:58.972847 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:59.255803 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:59.255724 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:26:59.709033 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:59.709002 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 06:26:59.709185 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:59.709051 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" podUID="a106c2d8-c5a5-4e22-b75d-922be5d0231a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 06:26:59.866749 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:26:59.866721 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:01.090814 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.090775 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:01.686511 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.686468 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b"] Oct 30 06:27:01.690506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.690482 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:01.710064 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.710035 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:27:01.710263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.710094 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" podUID="a106c2d8-c5a5-4e22-b75d-922be5d0231a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:27:01.715131 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.712331 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-6e2548-serving-cert\"" Oct 30 06:27:01.715131 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.715075 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b"] Oct 30 06:27:01.725511 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.725484 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76bkd\" (UniqueName: \"kubernetes.io/projected/67c137ac-abb6-412a-9dac-7639c95abaa0-kube-api-access-76bkd\") pod \"maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:01.725657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.725634 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:01.725746 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.725730 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:01.725898 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.725877 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67c137ac-abb6-412a-9dac-7639c95abaa0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:01.754154 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.754120 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:01.779309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.779281 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:27:01.779713 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.779681 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:27:01.780558 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.780540 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:27:01.827027 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.826996 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76bkd\" (UniqueName: \"kubernetes.io/projected/67c137ac-abb6-412a-9dac-7639c95abaa0-kube-api-access-76bkd\") pod \"maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:01.827253 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.827161 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:01.827334 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:27:01.827316 6999 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-serving-cert: secret "maas-api-e2e-shared-a-w2-6e2548-serving-cert" not found Oct 30 06:27:01.827334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.827322 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:01.827440 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:27:01.827383 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-maas-api-tls podName:67c137ac-abb6-412a-9dac-7639c95abaa0 nodeName:}" failed. No retries permitted until 2026-10-30 06:27:02.327361554 +0000 UTC m=+1988.836381436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-maas-api-tls") pod "maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" (UID: "67c137ac-abb6-412a-9dac-7639c95abaa0") : secret "maas-api-e2e-shared-a-w2-6e2548-serving-cert" not found Oct 30 06:27:01.827580 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.827555 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67c137ac-abb6-412a-9dac-7639c95abaa0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:01.828109 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.828087 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67c137ac-abb6-412a-9dac-7639c95abaa0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:01.829785 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.829768 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:01.851371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.851347 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76bkd\" (UniqueName: \"kubernetes.io/projected/67c137ac-abb6-412a-9dac-7639c95abaa0-kube-api-access-76bkd\") pod \"maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:01.911556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:01.911527 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:02.057946 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:02.057859 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:02.337859 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:02.337775 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:02.340467 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:02.340439 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:02.603157 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:02.603063 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:02.934553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:02.934521 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b"] Oct 30 06:27:02.964279 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:27:02.963581 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c137ac_abb6_412a_9dac_7639c95abaa0.slice/crio-08fcbbf14aa666d385685491c1e0830c74205e3f1ddae18a4bd2acf54cb853ef WatchSource:0}: Error finding container 08fcbbf14aa666d385685491c1e0830c74205e3f1ddae18a4bd2acf54cb853ef: Status 404 returned error can't find the container with id 08fcbbf14aa666d385685491c1e0830c74205e3f1ddae18a4bd2acf54cb853ef Oct 30 06:27:03.058288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:03.058255 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" event={"ID":"67c137ac-abb6-412a-9dac-7639c95abaa0","Type":"ContainerStarted","Data":"08fcbbf14aa666d385685491c1e0830c74205e3f1ddae18a4bd2acf54cb853ef"} Oct 30 06:27:03.234480 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:03.234382 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:03.857704 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:03.857671 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:04.074079 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:04.074038 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" event={"ID":"67c137ac-abb6-412a-9dac-7639c95abaa0","Type":"ContainerStarted","Data":"6f35d4a0ac5ab820010204fc9e4e4485893940e27bf6a633e64fa1c391d68f6e"} Oct 30 06:27:04.074295 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:04.074230 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:04.122168 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:04.122119 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" podStartSLOduration=2.7357440669999997 podStartE2EDuration="3.122104335s" totalImagesPullingTime="386.360268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:04.120458228 +0000 UTC m=+1990.629478114" watchObservedRunningTime="2026-10-30 06:27:04.122104335 +0000 UTC m=+1990.631124222" Oct 30 06:27:04.143080 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:27:04.143039 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:27:05.776082 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:05.776053 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:07.537304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:07.537266 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:07.910838 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:07.910806 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c"] Oct 30 06:27:07.940361 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:07.940330 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4"] Oct 30 06:27:08.026484 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:08.026454 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" Oct 30 06:27:08.089809 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:08.089777 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:08.244030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:08.243943 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:09.454369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:09.454330 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:09.682017 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:09.681986 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6"] Oct 30 06:27:09.682289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:09.682258 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" podUID="d9e281b2-b9c9-47a9-bb0a-96c313f43554" containerName="istio-proxy" containerID="cri-o://fddab5b5250d520413f9efc68dc7a12fa56282337a72792d4be614979d701647" gracePeriod=30 Oct 30 06:27:10.087796 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:10.087768 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:27:11.370800 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:11.370768 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:12.912099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:12.912044 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" podUID="1e816f9e-3d95-4347-9b8c-71146fba6add" containerName="payload-processing" containerID="cri-o://d409548e1b67c8343ea899391be1dec004e0494261412f8a1875245610b2dae7" gracePeriod=25 Oct 30 06:27:12.941670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:12.941629 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" podUID="058369d8-43cd-4de2-98f6-5fac727ec3e5" containerName="payload-pre-processing" containerID="cri-o://92544ff91c1bd65425d8b8cb3c21b2119598bb33dcc912889483bc769f8bc620" gracePeriod=25 Oct 30 06:27:13.089237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.084845 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964"] Oct 30 06:27:13.094257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.093458 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:13.143175 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.141479 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-fa5a69-serving-cert\"" Oct 30 06:27:13.143175 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.142046 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964"] Oct 30 06:27:13.147553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.147522 6999 generic.go:356] "Generic (PLEG): container finished" podID="058369d8-43cd-4de2-98f6-5fac727ec3e5" containerID="92544ff91c1bd65425d8b8cb3c21b2119598bb33dcc912889483bc769f8bc620" exitCode=0 Oct 30 06:27:13.147672 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.147622 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" event={"ID":"058369d8-43cd-4de2-98f6-5fac727ec3e5","Type":"ContainerDied","Data":"92544ff91c1bd65425d8b8cb3c21b2119598bb33dcc912889483bc769f8bc620"} Oct 30 06:27:13.149373 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.149347 6999 generic.go:356] "Generic (PLEG): container finished" podID="1e816f9e-3d95-4347-9b8c-71146fba6add" containerID="d409548e1b67c8343ea899391be1dec004e0494261412f8a1875245610b2dae7" exitCode=0 Oct 30 06:27:13.149492 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.149382 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" event={"ID":"1e816f9e-3d95-4347-9b8c-71146fba6add","Type":"ContainerDied","Data":"d409548e1b67c8343ea899391be1dec004e0494261412f8a1875245610b2dae7"} Oct 30 06:27:13.194126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.194088 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3ede4f-5a78-4223-837f-b0186e9be925-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:13.194416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.194382 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3ede4f-5a78-4223-837f-b0186e9be925-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:13.194540 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.194515 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3ede4f-5a78-4223-837f-b0186e9be925-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:13.194845 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.194821 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smzm5\" (UniqueName: \"kubernetes.io/projected/bb3ede4f-5a78-4223-837f-b0186e9be925-kube-api-access-smzm5\") pod \"maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:13.244123 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.244093 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" Oct 30 06:27:13.245730 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.245700 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:13.268611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.268590 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" Oct 30 06:27:13.296697 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.296661 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e816f9e-3d95-4347-9b8c-71146fba6add-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e816f9e-3d95-4347-9b8c-71146fba6add-plugins-config-volume\") pod \"1e816f9e-3d95-4347-9b8c-71146fba6add\" (UID: \"1e816f9e-3d95-4347-9b8c-71146fba6add\") " Oct 30 06:27:13.298304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.296737 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/058369d8-43cd-4de2-98f6-5fac727ec3e5-kube-api-access-mmh2n\" (UniqueName: \"kubernetes.io/projected/058369d8-43cd-4de2-98f6-5fac727ec3e5-kube-api-access-mmh2n\") pod \"058369d8-43cd-4de2-98f6-5fac727ec3e5\" (UID: \"058369d8-43cd-4de2-98f6-5fac727ec3e5\") " Oct 30 06:27:13.298304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.296870 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e816f9e-3d95-4347-9b8c-71146fba6add-kube-api-access-mr7sm\" (UniqueName: \"kubernetes.io/projected/1e816f9e-3d95-4347-9b8c-71146fba6add-kube-api-access-mr7sm\") pod \"1e816f9e-3d95-4347-9b8c-71146fba6add\" (UID: \"1e816f9e-3d95-4347-9b8c-71146fba6add\") " Oct 30 06:27:13.298304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.297009 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/058369d8-43cd-4de2-98f6-5fac727ec3e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058369d8-43cd-4de2-98f6-5fac727ec3e5-plugins-config-volume\") pod \"058369d8-43cd-4de2-98f6-5fac727ec3e5\" (UID: \"058369d8-43cd-4de2-98f6-5fac727ec3e5\") " Oct 30 06:27:13.298304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.297206 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3ede4f-5a78-4223-837f-b0186e9be925-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:13.298304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.297264 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3ede4f-5a78-4223-837f-b0186e9be925-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:13.298304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.297367 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3ede4f-5a78-4223-837f-b0186e9be925-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:13.298304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.297563 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smzm5\" (UniqueName: \"kubernetes.io/projected/bb3ede4f-5a78-4223-837f-b0186e9be925-kube-api-access-smzm5\") pod \"maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:13.298304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.298157 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e816f9e-3d95-4347-9b8c-71146fba6add-plugins-config-volume" pod "1e816f9e-3d95-4347-9b8c-71146fba6add" (UID: "1e816f9e-3d95-4347-9b8c-71146fba6add"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:27:13.300843 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.300819 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058369d8-43cd-4de2-98f6-5fac727ec3e5-kube-api-access-mmh2n" pod "058369d8-43cd-4de2-98f6-5fac727ec3e5" (UID: "058369d8-43cd-4de2-98f6-5fac727ec3e5"). InnerVolumeSpecName "kube-api-access-mmh2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:13.301451 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.301428 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3ede4f-5a78-4223-837f-b0186e9be925-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:13.301451 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.301444 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3ede4f-5a78-4223-837f-b0186e9be925-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:13.303388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.303366 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e816f9e-3d95-4347-9b8c-71146fba6add-kube-api-access-mr7sm" pod "1e816f9e-3d95-4347-9b8c-71146fba6add" (UID: "1e816f9e-3d95-4347-9b8c-71146fba6add"). InnerVolumeSpecName "kube-api-access-mr7sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:13.303568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.303547 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058369d8-43cd-4de2-98f6-5fac727ec3e5-plugins-config-volume" pod "058369d8-43cd-4de2-98f6-5fac727ec3e5" (UID: "058369d8-43cd-4de2-98f6-5fac727ec3e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:27:13.303657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.303646 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3ede4f-5a78-4223-837f-b0186e9be925-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:13.322925 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.322899 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smzm5\" (UniqueName: \"kubernetes.io/projected/bb3ede4f-5a78-4223-837f-b0186e9be925-kube-api-access-smzm5\") pod \"maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:13.399638 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.399591 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmh2n\" (UniqueName: \"kubernetes.io/projected/058369d8-43cd-4de2-98f6-5fac727ec3e5-kube-api-access-mmh2n\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:13.399638 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.399619 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr7sm\" (UniqueName: \"kubernetes.io/projected/1e816f9e-3d95-4347-9b8c-71146fba6add-kube-api-access-mr7sm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:13.399638 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.399636 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/058369d8-43cd-4de2-98f6-5fac727ec3e5-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:13.400191 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.399648 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e816f9e-3d95-4347-9b8c-71146fba6add-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:13.410944 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.410908 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:13.414800 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.414768 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:13.577570 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.577539 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:13.732199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:13.732169 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964"] Oct 30 06:27:13.733322 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:27:13.733298 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3ede4f_5a78_4223_837f_b0186e9be925.slice/crio-44f12ea9fa26d6e031a553919ee9e1b1d75e9620ab950b34b6020065fe600fad WatchSource:0}: Error finding container 44f12ea9fa26d6e031a553919ee9e1b1d75e9620ab950b34b6020065fe600fad: Status 404 returned error can't find the container with id 44f12ea9fa26d6e031a553919ee9e1b1d75e9620ab950b34b6020065fe600fad Oct 30 06:27:14.156881 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:14.156843 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" event={"ID":"bb3ede4f-5a78-4223-837f-b0186e9be925","Type":"ContainerStarted","Data":"5d9f6d7328557fceffc2000e474fe836033774ed1d5df3bf6a9de9bfa6c9b666"} Oct 30 06:27:14.157390 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:14.156906 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" event={"ID":"bb3ede4f-5a78-4223-837f-b0186e9be925","Type":"ContainerStarted","Data":"44f12ea9fa26d6e031a553919ee9e1b1d75e9620ab950b34b6020065fe600fad"} Oct 30 06:27:14.157390 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:14.157067 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:14.158483 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:14.158461 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" Oct 30 06:27:14.158596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:14.158464 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4" event={"ID":"058369d8-43cd-4de2-98f6-5fac727ec3e5","Type":"ContainerDied","Data":"d78f7d4b7f0146af9d2fc99d1e20e0b09fae20046535ed279980a4ae4c36d44f"} Oct 30 06:27:14.158596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:14.158587 6999 scope.go:122] "RemoveContainer" containerID="92544ff91c1bd65425d8b8cb3c21b2119598bb33dcc912889483bc769f8bc620" Oct 30 06:27:14.159913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:14.159893 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" event={"ID":"1e816f9e-3d95-4347-9b8c-71146fba6add","Type":"ContainerDied","Data":"c3f7ccdd82f513ed0f52580ccaccbed2463ead7d966022b70219b8f40468ba0f"} Oct 30 06:27:14.160026 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:14.159947 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c" Oct 30 06:27:14.168495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:14.168406 6999 scope.go:122] "RemoveContainer" containerID="d409548e1b67c8343ea899391be1dec004e0494261412f8a1875245610b2dae7" Oct 30 06:27:14.205024 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:14.204978 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" podStartSLOduration=0.933457791 podStartE2EDuration="1.204964491s" totalImagesPullingTime="271.5067ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:14.202708014 +0000 UTC m=+2000.711727901" watchObservedRunningTime="2026-10-30 06:27:14.204964491 +0000 UTC m=+2000.713984377" Oct 30 06:27:14.240727 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:14.240701 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4"] Oct 30 06:27:14.246918 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:14.246889 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ec27ed-866f5c6455-5hlz4"] Oct 30 06:27:14.279817 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:14.279738 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c"] Oct 30 06:27:14.285239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:14.285188 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ec27ed-5c6485859d-fxf9c"] Oct 30 06:27:14.962801 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:14.962736 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:27:15.020415 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.020379 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-workload-certs\") pod \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " Oct 30 06:27:15.020621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.020493 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-envoy\") pod \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " Oct 30 06:27:15.020621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.020536 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e281b2-b9c9-47a9-bb0a-96c313f43554-kube-api-access-swkt6\" (UniqueName: \"kubernetes.io/projected/d9e281b2-b9c9-47a9-bb0a-96c313f43554-kube-api-access-swkt6\") pod \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " Oct 30 06:27:15.020621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.020606 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-podinfo\") pod \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " Oct 30 06:27:15.020816 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.020662 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-workload-socket\") pod \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " Oct 30 06:27:15.020971 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.020943 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-workload-socket" pod "d9e281b2-b9c9-47a9-bb0a-96c313f43554" (UID: "d9e281b2-b9c9-47a9-bb0a-96c313f43554"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:15.021052 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.021006 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istiod-ca-cert\") pod \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " Oct 30 06:27:15.022141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.021131 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-data\") pod \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " Oct 30 06:27:15.022141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.021171 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-credential-socket\") pod \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " Oct 30 06:27:15.022141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.021266 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-token\" (UniqueName: \"kubernetes.io/projected/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-token\") pod \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\" (UID: \"d9e281b2-b9c9-47a9-bb0a-96c313f43554\") " Oct 30 06:27:15.022141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.021391 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istiod-ca-cert" pod "d9e281b2-b9c9-47a9-bb0a-96c313f43554" (UID: "d9e281b2-b9c9-47a9-bb0a-96c313f43554"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:27:15.022141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.021479 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-workload-certs" pod "d9e281b2-b9c9-47a9-bb0a-96c313f43554" (UID: "d9e281b2-b9c9-47a9-bb0a-96c313f43554"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:15.022141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.021562 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-data" pod "d9e281b2-b9c9-47a9-bb0a-96c313f43554" (UID: "d9e281b2-b9c9-47a9-bb0a-96c313f43554"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:15.022141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.021836 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-credential-socket" pod "d9e281b2-b9c9-47a9-bb0a-96c313f43554" (UID: "d9e281b2-b9c9-47a9-bb0a-96c313f43554"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:15.022141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.022057 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:15.022141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.022084 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:15.022141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.022099 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:15.022141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.022112 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:15.023505 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.023460 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-envoy" pod "d9e281b2-b9c9-47a9-bb0a-96c313f43554" (UID: "d9e281b2-b9c9-47a9-bb0a-96c313f43554"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:15.024137 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.024108 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e281b2-b9c9-47a9-bb0a-96c313f43554-kube-api-access-swkt6" pod "d9e281b2-b9c9-47a9-bb0a-96c313f43554" (UID: "d9e281b2-b9c9-47a9-bb0a-96c313f43554"). InnerVolumeSpecName "kube-api-access-swkt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:15.024409 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.024385 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-token" pod "d9e281b2-b9c9-47a9-bb0a-96c313f43554" (UID: "d9e281b2-b9c9-47a9-bb0a-96c313f43554"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:15.025319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.025300 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-podinfo" pod "d9e281b2-b9c9-47a9-bb0a-96c313f43554" (UID: "d9e281b2-b9c9-47a9-bb0a-96c313f43554"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:27:15.123624 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.123592 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:15.123624 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.123619 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:15.123624 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.123627 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:15.123847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.123635 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swkt6\" (UniqueName: \"kubernetes.io/projected/d9e281b2-b9c9-47a9-bb0a-96c313f43554-kube-api-access-swkt6\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:15.123847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.123645 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e281b2-b9c9-47a9-bb0a-96c313f43554-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:15.169348 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.169311 6999 generic.go:356] "Generic (PLEG): container finished" podID="d9e281b2-b9c9-47a9-bb0a-96c313f43554" containerID="fddab5b5250d520413f9efc68dc7a12fa56282337a72792d4be614979d701647" exitCode=0 Oct 30 06:27:15.169790 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.169360 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" Oct 30 06:27:15.169790 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.169399 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" event={"ID":"d9e281b2-b9c9-47a9-bb0a-96c313f43554","Type":"ContainerDied","Data":"fddab5b5250d520413f9efc68dc7a12fa56282337a72792d4be614979d701647"} Oct 30 06:27:15.169790 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.169474 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6" event={"ID":"d9e281b2-b9c9-47a9-bb0a-96c313f43554","Type":"ContainerDied","Data":"05ee5d560cd4498443a22b3da7fddd880d8c5e7d7f0eaf3ad6ec32fa8e0c305f"} Oct 30 06:27:15.169790 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.169501 6999 scope.go:122] "RemoveContainer" containerID="fddab5b5250d520413f9efc68dc7a12fa56282337a72792d4be614979d701647" Oct 30 06:27:15.180758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.180741 6999 scope.go:122] "RemoveContainer" containerID="fddab5b5250d520413f9efc68dc7a12fa56282337a72792d4be614979d701647" Oct 30 06:27:15.181023 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:27:15.181004 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fddab5b5250d520413f9efc68dc7a12fa56282337a72792d4be614979d701647\": container with ID starting with fddab5b5250d520413f9efc68dc7a12fa56282337a72792d4be614979d701647 not found: ID does not exist" containerID="fddab5b5250d520413f9efc68dc7a12fa56282337a72792d4be614979d701647" Oct 30 06:27:15.181116 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.181031 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fddab5b5250d520413f9efc68dc7a12fa56282337a72792d4be614979d701647"} err="failed to get container status \"fddab5b5250d520413f9efc68dc7a12fa56282337a72792d4be614979d701647\": rpc error: code = NotFound desc = could not find container \"fddab5b5250d520413f9efc68dc7a12fa56282337a72792d4be614979d701647\": container with ID starting with fddab5b5250d520413f9efc68dc7a12fa56282337a72792d4be614979d701647 not found: ID does not exist" Oct 30 06:27:15.222462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.222382 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6"] Oct 30 06:27:15.226997 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.226971 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-9s7l6"] Oct 30 06:27:15.263584 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.263557 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:15.609260 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.609208 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="058369d8-43cd-4de2-98f6-5fac727ec3e5" path="/var/lib/kubelet/pods/058369d8-43cd-4de2-98f6-5fac727ec3e5/volumes" Oct 30 06:27:15.609757 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.609732 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e816f9e-3d95-4347-9b8c-71146fba6add" path="/var/lib/kubelet/pods/1e816f9e-3d95-4347-9b8c-71146fba6add/volumes" Oct 30 06:27:15.610148 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:15.610136 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9e281b2-b9c9-47a9-bb0a-96c313f43554" path="/var/lib/kubelet/pods/d9e281b2-b9c9-47a9-bb0a-96c313f43554/volumes" Oct 30 06:27:16.920524 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:16.920487 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:17.291439 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:17.291350 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:17.454433 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:17.454406 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:17.995681 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:17.995643 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:18.310736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:18.310574 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:19.127680 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.127645 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z"] Oct 30 06:27:19.128459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.128435 6999 cpu_manager.go:414] "containerMap: removing container" podUID="058369d8-43cd-4de2-98f6-5fac727ec3e5" containerName="payload-pre-processing" Oct 30 06:27:19.128459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.128458 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="058369d8-43cd-4de2-98f6-5fac727ec3e5" containerName="payload-pre-processing" Oct 30 06:27:19.128630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.128478 6999 cpu_manager.go:414] "containerMap: removing container" podUID="d9e281b2-b9c9-47a9-bb0a-96c313f43554" containerName="istio-proxy" Oct 30 06:27:19.128630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.128484 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9e281b2-b9c9-47a9-bb0a-96c313f43554" containerName="istio-proxy" Oct 30 06:27:19.128630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.128502 6999 cpu_manager.go:414] "containerMap: removing container" podUID="1e816f9e-3d95-4347-9b8c-71146fba6add" containerName="payload-processing" Oct 30 06:27:19.128630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.128507 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e816f9e-3d95-4347-9b8c-71146fba6add" containerName="payload-processing" Oct 30 06:27:19.128630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.128594 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="058369d8-43cd-4de2-98f6-5fac727ec3e5" containerName="payload-pre-processing" Oct 30 06:27:19.128630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.128605 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9e281b2-b9c9-47a9-bb0a-96c313f43554" containerName="istio-proxy" Oct 30 06:27:19.128630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.128613 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e816f9e-3d95-4347-9b8c-71146fba6add" containerName="payload-processing" Oct 30 06:27:19.132882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.132865 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:19.147794 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.147775 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-4f3973-serving-cert\"" Oct 30 06:27:19.163333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.163309 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z"] Oct 30 06:27:19.174904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.174882 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5k4t\" (UniqueName: \"kubernetes.io/projected/3162d9da-8ad6-4272-8374-af59367f2884-kube-api-access-h5k4t\") pod \"maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:19.175166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.175135 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:19.175366 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.175282 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3162d9da-8ad6-4272-8374-af59367f2884-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:19.175366 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.175343 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:19.264457 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.264422 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:19.276993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.276969 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5k4t\" (UniqueName: \"kubernetes.io/projected/3162d9da-8ad6-4272-8374-af59367f2884-kube-api-access-h5k4t\") pod \"maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:19.277126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.277113 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:19.277174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.277144 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3162d9da-8ad6-4272-8374-af59367f2884-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:19.277242 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.277179 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:19.277448 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:27:19.277431 6999 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-serving-cert: secret "maas-api-e2e-shared-a-w0-4f3973-serving-cert" not found Oct 30 06:27:19.277551 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:27:19.277539 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-maas-api-tls podName:3162d9da-8ad6-4272-8374-af59367f2884 nodeName:}" failed. No retries permitted until 2026-10-30 06:27:19.777517736 +0000 UTC m=+2006.286537616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-maas-api-tls") pod "maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" (UID: "3162d9da-8ad6-4272-8374-af59367f2884") : secret "maas-api-e2e-shared-a-w0-4f3973-serving-cert" not found Oct 30 06:27:19.277979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.277948 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3162d9da-8ad6-4272-8374-af59367f2884-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:19.279613 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.279589 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:19.300798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.300770 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5k4t\" (UniqueName: \"kubernetes.io/projected/3162d9da-8ad6-4272-8374-af59367f2884-kube-api-access-h5k4t\") pod \"maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:19.412715 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.412617 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:19.621461 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.621411 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:19.784691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.784595 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:19.787243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:19.787198 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:20.046379 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:20.046206 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:20.177441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:20.177414 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 06:27:20.379852 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:27:20.379821 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3162d9da_8ad6_4272_8374_af59367f2884.slice/crio-d73dd240edc3e6e65a414fb60e53652f8d5c51ac8c033eda64a249b8bea6fada WatchSource:0}: Error finding container d73dd240edc3e6e65a414fb60e53652f8d5c51ac8c033eda64a249b8bea6fada: Status 404 returned error can't find the container with id d73dd240edc3e6e65a414fb60e53652f8d5c51ac8c033eda64a249b8bea6fada Oct 30 06:27:20.389279 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:20.383162 6999 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:27:20.393640 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:20.393611 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z"] Oct 30 06:27:21.218199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:21.218164 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" event={"ID":"3162d9da-8ad6-4272-8374-af59367f2884","Type":"ContainerStarted","Data":"ea71ee2213e1a04bc27d9726704b370337665662fd7e7966c013b37dc6d496b7"} Oct 30 06:27:21.218664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:21.218238 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" event={"ID":"3162d9da-8ad6-4272-8374-af59367f2884","Type":"ContainerStarted","Data":"d73dd240edc3e6e65a414fb60e53652f8d5c51ac8c033eda64a249b8bea6fada"} Oct 30 06:27:21.218664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:21.218260 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:21.263728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:21.263667 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" podStartSLOduration=1.881981271 podStartE2EDuration="2.263650208s" totalImagesPullingTime="381.668937ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:21.263153406 +0000 UTC m=+2007.772173295" watchObservedRunningTime="2026-10-30 06:27:21.263650208 +0000 UTC m=+2007.772670097" Oct 30 06:27:21.413967 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:21.413939 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:22.041776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:22.041743 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:22.226823 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:22.226791 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:23.551854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:23.551818 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:23.862356 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:23.862318 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:24.033890 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:24.033848 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:25.535602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:25.535568 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:27.019031 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.018995 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87"] Oct 30 06:27:27.019563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.019296 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" podUID="26c78c40-f1ad-46ea-bba5-807cdecde9c9" containerName="maas-api" containerID="cri-o://ce356b235445479ac16833ba28f7cd2a6d5a5b9df22f25da5bd99607a411cdb6" gracePeriod=30 Oct 30 06:27:27.158166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.158134 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:27.233290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.233260 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:27:27.263285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.263243 6999 generic.go:356] "Generic (PLEG): container finished" podID="26c78c40-f1ad-46ea-bba5-807cdecde9c9" containerID="ce356b235445479ac16833ba28f7cd2a6d5a5b9df22f25da5bd99607a411cdb6" exitCode=0 Oct 30 06:27:27.263517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.263496 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" event={"ID":"26c78c40-f1ad-46ea-bba5-807cdecde9c9","Type":"ContainerDied","Data":"ce356b235445479ac16833ba28f7cd2a6d5a5b9df22f25da5bd99607a411cdb6"} Oct 30 06:27:27.339282 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.339247 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:27:27.494559 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.494519 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26c78c40-f1ad-46ea-bba5-807cdecde9c9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c78c40-f1ad-46ea-bba5-807cdecde9c9-metrics-certs\") pod \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " Oct 30 06:27:27.494738 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.494583 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26c78c40-f1ad-46ea-bba5-807cdecde9c9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c78c40-f1ad-46ea-bba5-807cdecde9c9-maas-api-tls\") pod \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " Oct 30 06:27:27.494738 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.494643 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c78c40-f1ad-46ea-bba5-807cdecde9c9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c78c40-f1ad-46ea-bba5-807cdecde9c9-openshift-service-ca\") pod \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " Oct 30 06:27:27.494880 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.494752 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c78c40-f1ad-46ea-bba5-807cdecde9c9-kube-api-access-rx8lw\" (UniqueName: \"kubernetes.io/projected/26c78c40-f1ad-46ea-bba5-807cdecde9c9-kube-api-access-rx8lw\") pod \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\" (UID: \"26c78c40-f1ad-46ea-bba5-807cdecde9c9\") " Oct 30 06:27:27.495045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.495022 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c78c40-f1ad-46ea-bba5-807cdecde9c9-openshift-service-ca" pod "26c78c40-f1ad-46ea-bba5-807cdecde9c9" (UID: "26c78c40-f1ad-46ea-bba5-807cdecde9c9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:27:27.495277 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.495256 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/26c78c40-f1ad-46ea-bba5-807cdecde9c9-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:27.496978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.496949 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c78c40-f1ad-46ea-bba5-807cdecde9c9-metrics-certs" pod "26c78c40-f1ad-46ea-bba5-807cdecde9c9" (UID: "26c78c40-f1ad-46ea-bba5-807cdecde9c9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:27:27.496978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.496956 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c78c40-f1ad-46ea-bba5-807cdecde9c9-kube-api-access-rx8lw" pod "26c78c40-f1ad-46ea-bba5-807cdecde9c9" (UID: "26c78c40-f1ad-46ea-bba5-807cdecde9c9"). InnerVolumeSpecName "kube-api-access-rx8lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:27.496978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.496968 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c78c40-f1ad-46ea-bba5-807cdecde9c9-maas-api-tls" pod "26c78c40-f1ad-46ea-bba5-807cdecde9c9" (UID: "26c78c40-f1ad-46ea-bba5-807cdecde9c9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:27:27.597237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.597097 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26c78c40-f1ad-46ea-bba5-807cdecde9c9-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:27.597467 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.597442 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/26c78c40-f1ad-46ea-bba5-807cdecde9c9-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:27.597563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.597467 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx8lw\" (UniqueName: \"kubernetes.io/projected/26c78c40-f1ad-46ea-bba5-807cdecde9c9-kube-api-access-rx8lw\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:27:27.757430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.757400 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:27.937362 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:27.937334 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:28.271755 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:28.271675 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" Oct 30 06:27:28.271755 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:28.271708 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87" event={"ID":"26c78c40-f1ad-46ea-bba5-807cdecde9c9","Type":"ContainerDied","Data":"e310c9dd35918fd822d852efab53bc9cb3ef79f641325e1b5f6c2d1b3d43772d"} Oct 30 06:27:28.272238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:28.271772 6999 scope.go:122] "RemoveContainer" containerID="ce356b235445479ac16833ba28f7cd2a6d5a5b9df22f25da5bd99607a411cdb6" Oct 30 06:27:28.328508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:28.328481 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87"] Oct 30 06:27:28.340760 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:28.340720 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ec27ed-847c746bb8-fpv87"] Oct 30 06:27:28.424013 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:28.423985 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:29.608801 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:29.608770 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c78c40-f1ad-46ea-bba5-807cdecde9c9" path="/var/lib/kubelet/pods/26c78c40-f1ad-46ea-bba5-807cdecde9c9/volumes" Oct 30 06:27:29.776769 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:29.776737 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:31.175449 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:31.175409 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:31.708779 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:31.708731 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:32.010597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:32.010510 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:32.310297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:32.305888 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:32.488727 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:32.488690 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:33.829889 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:33.829856 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:34.142537 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:27:34.142497 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:27:35.432525 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:35.432454 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:35.587936 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:35.587897 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:35.794821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:35.794737 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:35.979195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:35.979151 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:36.164497 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:36.164460 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:36.351873 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:36.351840 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:37.868042 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:37.868010 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:38.018942 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.018906 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:38.023435 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.023374 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7"] Oct 30 06:27:38.024150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.024125 6999 cpu_manager.go:414] "containerMap: removing container" podUID="26c78c40-f1ad-46ea-bba5-807cdecde9c9" containerName="maas-api" Oct 30 06:27:38.024308 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.024286 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c78c40-f1ad-46ea-bba5-807cdecde9c9" containerName="maas-api" Oct 30 06:27:38.024490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.024479 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c78c40-f1ad-46ea-bba5-807cdecde9c9" containerName="maas-api" Oct 30 06:27:38.027866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.027843 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" Oct 30 06:27:38.038821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.038800 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8e33ed-9831-4c77-8fdd-b64f9a593b99-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7\" (UID: \"fc8e33ed-9831-4c77-8fdd-b64f9a593b99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" Oct 30 06:27:38.039047 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.039030 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r77k\" (UniqueName: \"kubernetes.io/projected/fc8e33ed-9831-4c77-8fdd-b64f9a593b99-kube-api-access-6r77k\") pod \"payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7\" (UID: \"fc8e33ed-9831-4c77-8fdd-b64f9a593b99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" Oct 30 06:27:38.056466 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.056449 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-868b1f-dockercfg-chlzb\"" Oct 30 06:27:38.060836 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.060769 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7"] Oct 30 06:27:38.105335 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.092545 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-868b1f\"" Oct 30 06:27:38.145289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.140388 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r77k\" (UniqueName: \"kubernetes.io/projected/fc8e33ed-9831-4c77-8fdd-b64f9a593b99-kube-api-access-6r77k\") pod \"payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7\" (UID: \"fc8e33ed-9831-4c77-8fdd-b64f9a593b99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" Oct 30 06:27:38.145289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.140501 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8e33ed-9831-4c77-8fdd-b64f9a593b99-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7\" (UID: \"fc8e33ed-9831-4c77-8fdd-b64f9a593b99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" Oct 30 06:27:38.145289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.141398 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8e33ed-9831-4c77-8fdd-b64f9a593b99-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7\" (UID: \"fc8e33ed-9831-4c77-8fdd-b64f9a593b99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" Oct 30 06:27:38.164452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.164338 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r77k\" (UniqueName: \"kubernetes.io/projected/fc8e33ed-9831-4c77-8fdd-b64f9a593b99-kube-api-access-6r77k\") pod \"payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7\" (UID: \"fc8e33ed-9831-4c77-8fdd-b64f9a593b99\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" Oct 30 06:27:38.341481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.341450 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" Oct 30 06:27:38.670495 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:27:38.670467 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8e33ed_9831_4c77_8fdd_b64f9a593b99.slice/crio-59a0a1c8dc65864e9ca45ab361eb423ff16ddeaaee54e49d4a95b2626d79d8b9 WatchSource:0}: Error finding container 59a0a1c8dc65864e9ca45ab361eb423ff16ddeaaee54e49d4a95b2626d79d8b9: Status 404 returned error can't find the container with id 59a0a1c8dc65864e9ca45ab361eb423ff16ddeaaee54e49d4a95b2626d79d8b9 Oct 30 06:27:38.671111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:38.671035 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7"] Oct 30 06:27:39.351854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:39.351816 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" event={"ID":"fc8e33ed-9831-4c77-8fdd-b64f9a593b99","Type":"ContainerStarted","Data":"88d16e753a568ded154fb0bce97634a071c074a264845c5f6033c3220bc92eb4"} Oct 30 06:27:39.352310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:39.351903 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" event={"ID":"fc8e33ed-9831-4c77-8fdd-b64f9a593b99","Type":"ContainerStarted","Data":"59a0a1c8dc65864e9ca45ab361eb423ff16ddeaaee54e49d4a95b2626d79d8b9"} Oct 30 06:27:39.352310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:39.352035 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" Oct 30 06:27:39.392249 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:39.392178 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" podStartSLOduration=2.392165211 podStartE2EDuration="2.392165211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:39.391746229 +0000 UTC m=+2025.900766116" watchObservedRunningTime="2026-10-30 06:27:39.392165211 +0000 UTC m=+2025.901185098" Oct 30 06:27:40.003203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:40.003163 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:41.999732 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:41.999700 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:43.386848 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.386812 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:43.665816 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.665734 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx"] Oct 30 06:27:43.670126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.670105 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:43.703354 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.703322 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-30fbb6-serving-cert\"" Oct 30 06:27:43.704024 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.703993 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx"] Oct 30 06:27:43.711107 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.711083 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d5ac7b-9102-4ecb-bb87-53182e8898d2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:43.711366 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.711328 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2mz4\" (UniqueName: \"kubernetes.io/projected/51d5ac7b-9102-4ecb-bb87-53182e8898d2-kube-api-access-c2mz4\") pod \"maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:43.711494 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.711399 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51d5ac7b-9102-4ecb-bb87-53182e8898d2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:43.711703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.711671 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51d5ac7b-9102-4ecb-bb87-53182e8898d2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:43.795882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.795858 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:43.812624 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.812594 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2mz4\" (UniqueName: \"kubernetes.io/projected/51d5ac7b-9102-4ecb-bb87-53182e8898d2-kube-api-access-c2mz4\") pod \"maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:43.812739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.812632 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51d5ac7b-9102-4ecb-bb87-53182e8898d2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:43.812739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.812732 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51d5ac7b-9102-4ecb-bb87-53182e8898d2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:43.812935 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.812770 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d5ac7b-9102-4ecb-bb87-53182e8898d2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:43.813520 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.813496 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d5ac7b-9102-4ecb-bb87-53182e8898d2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:43.815241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.815204 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51d5ac7b-9102-4ecb-bb87-53182e8898d2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:43.815435 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.815414 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51d5ac7b-9102-4ecb-bb87-53182e8898d2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:43.836132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.836103 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2mz4\" (UniqueName: \"kubernetes.io/projected/51d5ac7b-9102-4ecb-bb87-53182e8898d2-kube-api-access-c2mz4\") pod \"maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:43.947031 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.946948 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:43.983859 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:43.983829 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:44.292739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:44.292713 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx"] Oct 30 06:27:44.294133 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:27:44.294097 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d5ac7b_9102_4ecb_bb87_53182e8898d2.slice/crio-43c3d5377b18fbf748f7308be4e7d31c5eded130a35608124ea4c250f5fcb95d WatchSource:0}: Error finding container 43c3d5377b18fbf748f7308be4e7d31c5eded130a35608124ea4c250f5fcb95d: Status 404 returned error can't find the container with id 43c3d5377b18fbf748f7308be4e7d31c5eded130a35608124ea4c250f5fcb95d Oct 30 06:27:44.384851 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:44.384819 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" event={"ID":"51d5ac7b-9102-4ecb-bb87-53182e8898d2","Type":"ContainerStarted","Data":"43c3d5377b18fbf748f7308be4e7d31c5eded130a35608124ea4c250f5fcb95d"} Oct 30 06:27:45.392659 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.392619 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" event={"ID":"51d5ac7b-9102-4ecb-bb87-53182e8898d2","Type":"ContainerStarted","Data":"fab04c8149922bdb44bea2840b56ecfabd540a2685e10641cdea7d745f2e28ce"} Oct 30 06:27:45.393061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.392740 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:45.594990 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.594963 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:45.648665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.648571 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" podStartSLOduration=2.343756107 podStartE2EDuration="2.648556334s" totalImagesPullingTime="304.800227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:45.437462853 +0000 UTC m=+2031.946482744" watchObservedRunningTime="2026-10-30 06:27:45.648556334 +0000 UTC m=+2032.157576222" Oct 30 06:27:45.649974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.649913 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz"] Oct 30 06:27:45.654199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.654179 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.678527 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.678501 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-adcdcb\"/\"openshift-service-ca.crt\"" Oct 30 06:27:45.678527 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.678517 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-adcdcb\"/\"kube-root-ca.crt\"" Oct 30 06:27:45.678712 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.678672 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-adcdcb\"/\"auth-test-model-adcdcb-kserve-self-signed-certs\"" Oct 30 06:27:45.678771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.678722 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-adcdcb\"/\"default-dockercfg-x89j7\"" Oct 30 06:27:45.682137 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.682105 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz"] Oct 30 06:27:45.777782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.777753 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:45.781171 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.781146 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-tmp-dir\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.781322 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.781200 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87001573-6955-4231-9fb9-93e335b7d16b-tls-certs\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.781364 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.781334 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7vz9\" (UniqueName: \"kubernetes.io/projected/87001573-6955-4231-9fb9-93e335b7d16b-kube-api-access-q7vz9\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.781425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.781406 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-dshm\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.781462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.781444 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-model-cache\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.781614 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.781601 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-home\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.882699 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.882663 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-dshm\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.882903 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.882717 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-model-cache\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.882985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.882908 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-home\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.882985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.882979 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-tmp-dir\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.883108 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.883048 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87001573-6955-4231-9fb9-93e335b7d16b-tls-certs\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.883273 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.883199 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-model-cache\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.883461 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.883442 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-home\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.883533 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.883497 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7vz9\" (UniqueName: \"kubernetes.io/projected/87001573-6955-4231-9fb9-93e335b7d16b-kube-api-access-q7vz9\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.883533 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.883519 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-tmp-dir\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.885642 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.885608 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-dshm\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.886456 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.886385 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87001573-6955-4231-9fb9-93e335b7d16b-tls-certs\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.907481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.907408 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7vz9\" (UniqueName: \"kubernetes.io/projected/87001573-6955-4231-9fb9-93e335b7d16b-kube-api-access-q7vz9\") pod \"auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:45.921825 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.921608 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:45.976831 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:45.976791 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:46.084418 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:46.084383 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:46.314810 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:46.314725 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz"] Oct 30 06:27:46.319054 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:27:46.319026 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87001573_6955_4231_9fb9_93e335b7d16b.slice/crio-bcca1145f156eff915fc688fb3af6afa08e4ce372e09dd14820ae5d831b1ef00 WatchSource:0}: Error finding container bcca1145f156eff915fc688fb3af6afa08e4ce372e09dd14820ae5d831b1ef00: Status 404 returned error can't find the container with id bcca1145f156eff915fc688fb3af6afa08e4ce372e09dd14820ae5d831b1ef00 Oct 30 06:27:46.405698 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:46.405663 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" event={"ID":"87001573-6955-4231-9fb9-93e335b7d16b","Type":"ContainerStarted","Data":"8a778b2dd24f90e8ac0fb128cadd536a3289a4c603f5cf235c62c449038853d6"} Oct 30 06:27:46.406086 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:46.405712 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" event={"ID":"87001573-6955-4231-9fb9-93e335b7d16b","Type":"ContainerStarted","Data":"bcca1145f156eff915fc688fb3af6afa08e4ce372e09dd14820ae5d831b1ef00"} Oct 30 06:27:46.445000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:46.444935 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:46.451334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:46.451285 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" podStartSLOduration=1.451264742 podStartE2EDuration="1.451264742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:46.449335971 +0000 UTC m=+2032.958355859" watchObservedRunningTime="2026-10-30 06:27:46.451264742 +0000 UTC m=+2032.960284630" Oct 30 06:27:46.977752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:46.977705 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:47.700859 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:47.700810 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:48.173571 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:48.173534 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:48.658548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:48.658497 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:49.131820 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:49.131783 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:49.298702 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:49.298670 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:49.530640 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:49.530546 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:50.362014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:50.361989 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" Oct 30 06:27:50.606613 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:50.606578 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:50.768145 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:50.768020 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:51.412059 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:51.412033 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 06:27:51.643014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:51.642977 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:52.412929 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:52.412897 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:52.585472 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:52.585439 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:52.829300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:52.827243 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:53.819277 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:53.819202 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:54.386238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:54.386160 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:54.571029 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:54.570992 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:55.932521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:55.932480 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:55.978195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:55.978161 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:55.991996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:55.991966 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:56.491737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:56.491702 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:27:57.731736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:57.731702 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:59.302128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.302089 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85"] Oct 30 06:27:59.307155 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.307135 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:27:59.327389 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.327360 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-868b1f-serving-cert\"" Oct 30 06:27:59.344443 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.344413 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85"] Oct 30 06:27:59.378660 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.378625 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/701a7be4-531f-434e-9e1e-a85db672c13f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:27:59.378839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.378810 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/701a7be4-531f-434e-9e1e-a85db672c13f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:27:59.378923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.378905 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kznk\" (UniqueName: \"kubernetes.io/projected/701a7be4-531f-434e-9e1e-a85db672c13f-kube-api-access-7kznk\") pod \"maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:27:59.378976 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.378961 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/701a7be4-531f-434e-9e1e-a85db672c13f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:27:59.422582 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.422552 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:59.480145 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.480100 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/701a7be4-531f-434e-9e1e-a85db672c13f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:27:59.480408 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.480203 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/701a7be4-531f-434e-9e1e-a85db672c13f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:27:59.480408 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.480280 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kznk\" (UniqueName: \"kubernetes.io/projected/701a7be4-531f-434e-9e1e-a85db672c13f-kube-api-access-7kznk\") pod \"maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:27:59.480408 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.480318 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/701a7be4-531f-434e-9e1e-a85db672c13f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:27:59.481058 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.481027 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/701a7be4-531f-434e-9e1e-a85db672c13f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:27:59.482991 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.482967 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/701a7be4-531f-434e-9e1e-a85db672c13f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:27:59.483131 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.483110 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/701a7be4-531f-434e-9e1e-a85db672c13f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:27:59.504764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.504730 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kznk\" (UniqueName: \"kubernetes.io/projected/701a7be4-531f-434e-9e1e-a85db672c13f-kube-api-access-7kznk\") pod \"maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:27:59.571310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.571197 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:27:59.621613 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.621575 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:27:59.940204 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:27:59.940181 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85"] Oct 30 06:27:59.945255 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:27:59.942007 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701a7be4_531f_434e_9e1e_a85db672c13f.slice/crio-a35f91f3cd586ee9811a5042afc15881fcfbcec6d7c9e34d2f4a636b88faba33 WatchSource:0}: Error finding container a35f91f3cd586ee9811a5042afc15881fcfbcec6d7c9e34d2f4a636b88faba33: Status 404 returned error can't find the container with id a35f91f3cd586ee9811a5042afc15881fcfbcec6d7c9e34d2f4a636b88faba33 Oct 30 06:28:00.508237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:00.508191 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" event={"ID":"701a7be4-531f-434e-9e1e-a85db672c13f","Type":"ContainerStarted","Data":"806a2456ccab2a55ffa9b335b6c11aa96f351de2e665a8a9023502ac4debe58f"} Oct 30 06:28:00.508623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:00.508282 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:28:00.508623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:00.508305 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" event={"ID":"701a7be4-531f-434e-9e1e-a85db672c13f","Type":"ContainerStarted","Data":"a35f91f3cd586ee9811a5042afc15881fcfbcec6d7c9e34d2f4a636b88faba33"} Oct 30 06:28:00.553054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:00.553001 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" podStartSLOduration=1.139138593 podStartE2EDuration="1.55298572s" totalImagesPullingTime="413.847127ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:28:00.550115875 +0000 UTC m=+2047.059135771" watchObservedRunningTime="2026-10-30 06:28:00.55298572 +0000 UTC m=+2047.062005606" Oct 30 06:28:01.230078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:01.230045 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:01.426882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:01.426853 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:03.334190 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:03.334155 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:03.483602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:03.483569 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:04.142269 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:28:04.142229 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:28:05.246510 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:05.246475 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:06.520113 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:06.520083 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:28:07.229139 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:07.229101 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:07.995856 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:07.995817 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95"] Oct 30 06:28:08.001098 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.001076 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.016978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.016939 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-fa5a69\"/\"default-dockercfg-kh8sz\"" Oct 30 06:28:08.029320 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.029290 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-fa5a69\"/\"auth-test-model-fa5a69-kserve-self-signed-certs\"" Oct 30 06:28:08.029766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.029742 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-fa5a69\"/\"kube-root-ca.crt\"" Oct 30 06:28:08.032651 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.032616 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-fa5a69\"/\"openshift-service-ca.crt\"" Oct 30 06:28:08.037085 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.037059 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95"] Oct 30 06:28:08.096360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.096325 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:08.103691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.103658 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-model-cache\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.103845 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.103759 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-tmp-dir\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.103845 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.103823 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1af9cf-0a72-4d17-9544-c1e86f081185-tls-certs\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.103919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.103858 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-dshm\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.103919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.103906 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-home\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.103984 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.103949 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smsqg\" (UniqueName: \"kubernetes.io/projected/dd1af9cf-0a72-4d17-9544-c1e86f081185-kube-api-access-smsqg\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.204964 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.204928 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-model-cache\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.205176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.204988 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-tmp-dir\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.205176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.205040 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1af9cf-0a72-4d17-9544-c1e86f081185-tls-certs\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.205176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.205073 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-dshm\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.205176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.205125 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-home\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.205176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.205173 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smsqg\" (UniqueName: \"kubernetes.io/projected/dd1af9cf-0a72-4d17-9544-c1e86f081185-kube-api-access-smsqg\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.205820 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.205464 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-model-cache\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.205820 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.205486 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-tmp-dir\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.205820 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.205529 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-home\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.207770 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.207746 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-dshm\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.208000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.207981 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1af9cf-0a72-4d17-9544-c1e86f081185-tls-certs\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.229444 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.229411 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smsqg\" (UniqueName: \"kubernetes.io/projected/dd1af9cf-0a72-4d17-9544-c1e86f081185-kube-api-access-smsqg\") pod \"auth-test-model-fa5a69-kserve-56d444fdb9-xtp95\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.317483 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.317051 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:08.638248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:08.638192 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95"] Oct 30 06:28:08.639688 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:28:08.639661 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd1af9cf_0a72_4d17_9544_c1e86f081185.slice/crio-aa17401aa71362f8819c5c30d10dd538f0b6f983a41124cf209719e42ffab39e WatchSource:0}: Error finding container aa17401aa71362f8819c5c30d10dd538f0b6f983a41124cf209719e42ffab39e: Status 404 returned error can't find the container with id aa17401aa71362f8819c5c30d10dd538f0b6f983a41124cf209719e42ffab39e Oct 30 06:28:09.039781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:09.039693 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:09.566634 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:09.566593 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" event={"ID":"dd1af9cf-0a72-4d17-9544-c1e86f081185","Type":"ContainerStarted","Data":"51b3d8394b7264d202c202b862963391bb107efeaa3a434e7d5266c417480f23"} Oct 30 06:28:09.566826 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:09.566645 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" event={"ID":"dd1af9cf-0a72-4d17-9544-c1e86f081185","Type":"ContainerStarted","Data":"aa17401aa71362f8819c5c30d10dd538f0b6f983a41124cf209719e42ffab39e"} Oct 30 06:28:09.609643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:09.609595 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" podStartSLOduration=2.609581036 podStartE2EDuration="2.609581036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:28:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:28:09.607835307 +0000 UTC m=+2056.116855195" watchObservedRunningTime="2026-10-30 06:28:09.609581036 +0000 UTC m=+2056.118600944" Oct 30 06:28:10.317897 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:10.317860 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:10.826896 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:10.826859 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:12.734872 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:12.734833 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:13.012907 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:13.012827 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:14.519206 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.519174 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg"] Oct 30 06:28:14.523648 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.523629 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:14.544651 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.544621 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg"] Oct 30 06:28:14.552832 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.552805 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-3bbfc8-serving-cert\"" Oct 30 06:28:14.645455 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.645423 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:14.703919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.703881 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0aee967-261a-42ce-9e34-2d62082786ed-maas-api-tls\") pod \"maas-api-e2e-explicit-3bbfc8-df989896-v94cg\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:14.704155 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.704133 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0aee967-261a-42ce-9e34-2d62082786ed-openshift-service-ca\") pod \"maas-api-e2e-explicit-3bbfc8-df989896-v94cg\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:14.704312 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.704298 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s724\" (UniqueName: \"kubernetes.io/projected/e0aee967-261a-42ce-9e34-2d62082786ed-kube-api-access-4s724\") pod \"maas-api-e2e-explicit-3bbfc8-df989896-v94cg\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:14.704379 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.704367 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0aee967-261a-42ce-9e34-2d62082786ed-metrics-certs\") pod \"maas-api-e2e-explicit-3bbfc8-df989896-v94cg\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:14.804983 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.804881 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:14.806203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.805988 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0aee967-261a-42ce-9e34-2d62082786ed-openshift-service-ca\") pod \"maas-api-e2e-explicit-3bbfc8-df989896-v94cg\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:14.806746 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.806717 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s724\" (UniqueName: \"kubernetes.io/projected/e0aee967-261a-42ce-9e34-2d62082786ed-kube-api-access-4s724\") pod \"maas-api-e2e-explicit-3bbfc8-df989896-v94cg\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:14.807014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.806841 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0aee967-261a-42ce-9e34-2d62082786ed-metrics-certs\") pod \"maas-api-e2e-explicit-3bbfc8-df989896-v94cg\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:14.807014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.806885 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0aee967-261a-42ce-9e34-2d62082786ed-maas-api-tls\") pod \"maas-api-e2e-explicit-3bbfc8-df989896-v94cg\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:14.807014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.807009 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0aee967-261a-42ce-9e34-2d62082786ed-openshift-service-ca\") pod \"maas-api-e2e-explicit-3bbfc8-df989896-v94cg\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:14.810546 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.810517 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0aee967-261a-42ce-9e34-2d62082786ed-metrics-certs\") pod \"maas-api-e2e-explicit-3bbfc8-df989896-v94cg\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:14.812860 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.812838 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0aee967-261a-42ce-9e34-2d62082786ed-maas-api-tls\") pod \"maas-api-e2e-explicit-3bbfc8-df989896-v94cg\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:14.830439 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.830414 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s724\" (UniqueName: \"kubernetes.io/projected/e0aee967-261a-42ce-9e34-2d62082786ed-kube-api-access-4s724\") pod \"maas-api-e2e-explicit-3bbfc8-df989896-v94cg\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:14.838304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.838275 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:14.952847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:14.952816 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:15.150644 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:15.150610 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg"] Oct 30 06:28:15.151785 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:28:15.151759 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0aee967_261a_42ce_9e34_2d62082786ed.slice/crio-c34c57f9fb65e7d95d8a37113155b6ed80e3175eea782535566e754b113ce73d WatchSource:0}: Error finding container c34c57f9fb65e7d95d8a37113155b6ed80e3175eea782535566e754b113ce73d: Status 404 returned error can't find the container with id c34c57f9fb65e7d95d8a37113155b6ed80e3175eea782535566e754b113ce73d Oct 30 06:28:15.645512 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:15.645472 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" event={"ID":"e0aee967-261a-42ce-9e34-2d62082786ed","Type":"ContainerStarted","Data":"d092ac43acb07cdd7b9a95b6dd6f06b2d26e50333ac23227df00bbc02423cc03"} Oct 30 06:28:15.645985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:15.645542 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" event={"ID":"e0aee967-261a-42ce-9e34-2d62082786ed","Type":"ContainerStarted","Data":"c34c57f9fb65e7d95d8a37113155b6ed80e3175eea782535566e754b113ce73d"} Oct 30 06:28:15.645985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:15.645572 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:15.690193 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:15.690146 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" podStartSLOduration=1.40913192 podStartE2EDuration="1.690132092s" totalImagesPullingTime="281.000172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:28:15.687483511 +0000 UTC m=+2062.196503496" watchObservedRunningTime="2026-10-30 06:28:15.690132092 +0000 UTC m=+2062.199151971" Oct 30 06:28:16.607182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:16.607119 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:17.561534 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:17.561500 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:17.709414 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:17.709364 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:18.267049 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.267014 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj"] Oct 30 06:28:18.271350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.271328 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:18.291094 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.291056 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-b9ff93-serving-cert\"" Oct 30 06:28:18.305176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.305141 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj"] Oct 30 06:28:18.318405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.318375 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:18.335078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.335053 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:18.361396 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.361363 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr9dr\" (UniqueName: \"kubernetes.io/projected/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-kube-api-access-tr9dr\") pod \"maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:18.361608 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.361447 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:18.361804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.361781 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:18.361862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.361835 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:18.390987 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.390950 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:18.463003 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.462968 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:18.463259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.463007 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:18.463259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.463096 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr9dr\" (UniqueName: \"kubernetes.io/projected/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-kube-api-access-tr9dr\") pod \"maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:18.463259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.463184 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:18.463783 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.463758 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:18.465812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.465791 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:18.465898 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.465829 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:18.486788 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.486763 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr9dr\" (UniqueName: \"kubernetes.io/projected/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-kube-api-access-tr9dr\") pod \"maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:18.542026 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.534575 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:18.586559 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.586522 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:18.690031 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.688868 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:18.714264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.714235 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:28:18.901566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:18.901535 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj"] Oct 30 06:28:18.902983 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:28:18.902955 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4c6e9e_70db_47c3_82f5_e1798c4e6009.slice/crio-aecdd7850cedbf7886d3ee08c4b6fd74d1f97c0b66842a5673331a26e7b70779 WatchSource:0}: Error finding container aecdd7850cedbf7886d3ee08c4b6fd74d1f97c0b66842a5673331a26e7b70779: Status 404 returned error can't find the container with id aecdd7850cedbf7886d3ee08c4b6fd74d1f97c0b66842a5673331a26e7b70779 Oct 30 06:28:19.703860 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:19.703812 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" event={"ID":"4f4c6e9e-70db-47c3-82f5-e1798c4e6009","Type":"ContainerStarted","Data":"1234af9cdf44f98256e88e99f7802582d988f0524cc6c8508082b4848c5ac0bc"} Oct 30 06:28:19.704399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:19.703885 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" event={"ID":"4f4c6e9e-70db-47c3-82f5-e1798c4e6009","Type":"ContainerStarted","Data":"aecdd7850cedbf7886d3ee08c4b6fd74d1f97c0b66842a5673331a26e7b70779"} Oct 30 06:28:19.747323 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:19.747264 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" podStartSLOduration=1.398604683 podStartE2EDuration="1.747247361s" totalImagesPullingTime="348.642678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:28:19.744911576 +0000 UTC m=+2066.253931669" watchObservedRunningTime="2026-10-30 06:28:19.747247361 +0000 UTC m=+2066.256267267" Oct 30 06:28:20.269910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:20.269878 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:20.715768 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:20.715732 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:21.658325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:21.658293 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:28:22.116111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:22.116081 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:23.931528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:23.931484 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:25.692740 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:25.692710 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:25.820236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:25.819650 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj"] Oct 30 06:28:25.828240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:25.825665 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:25.832126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:25.829845 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj"] Oct 30 06:28:25.854399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:25.854361 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-645f2b\"/\"default-dockercfg-gcr6r\"" Oct 30 06:28:25.854399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:25.854361 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-645f2b\"/\"auth-test-model-645f2b-kserve-self-signed-certs\"" Oct 30 06:28:25.854635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:25.854406 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-645f2b\"/\"openshift-service-ca.crt\"" Oct 30 06:28:25.854635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:25.854378 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-645f2b\"/\"kube-root-ca.crt\"" Oct 30 06:28:25.893057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:25.893029 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:25.917730 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:25.917694 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-home\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:25.917906 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:25.917880 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-dshm\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:25.917961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:25.917934 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zxd4\" (UniqueName: \"kubernetes.io/projected/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-kube-api-access-8zxd4\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:25.918165 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:25.918147 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-tls-certs\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:25.918204 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:25.918180 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-tmp-dir\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:25.918388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:25.918360 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-model-cache\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:26.019819 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.019718 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zxd4\" (UniqueName: \"kubernetes.io/projected/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-kube-api-access-8zxd4\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:26.020003 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.019868 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-tls-certs\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:26.020003 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.019892 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-tmp-dir\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:26.020003 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.019969 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-model-cache\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:26.020188 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.020003 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-home\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:26.020188 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.020118 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-dshm\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:26.021060 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.021032 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-model-cache\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:26.021283 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.021086 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-home\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:26.021471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.021119 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-tmp-dir\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:26.026504 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.026470 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-dshm\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:26.050887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.050850 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-tls-certs\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:26.052699 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.052643 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zxd4\" (UniqueName: \"kubernetes.io/projected/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-kube-api-access-8zxd4\") pod \"auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:26.146002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.145968 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:26.471922 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.471874 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj"] Oct 30 06:28:26.483918 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:28:26.483886 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3d4a9bc_8241_4e6b_8af0_2ada17fffceb.slice/crio-adc1e813565c861778314bb5ad51a2bbe790e302ffdbd7aa48ac8ef1a18efae7 WatchSource:0}: Error finding container adc1e813565c861778314bb5ad51a2bbe790e302ffdbd7aa48ac8ef1a18efae7: Status 404 returned error can't find the container with id adc1e813565c861778314bb5ad51a2bbe790e302ffdbd7aa48ac8ef1a18efae7 Oct 30 06:28:26.730921 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.730845 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:28:26.759398 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.759366 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" event={"ID":"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb","Type":"ContainerStarted","Data":"3d464df6b8e2ef581182415edc30277ff03545f27d67788636771813e5b70b7d"} Oct 30 06:28:26.759555 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.759408 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" event={"ID":"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb","Type":"ContainerStarted","Data":"adc1e813565c861778314bb5ad51a2bbe790e302ffdbd7aa48ac8ef1a18efae7"} Oct 30 06:28:26.820565 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:26.820498 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" podStartSLOduration=1.8204770049999999 podStartE2EDuration="1.820477005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:28:26.813275241 +0000 UTC m=+2073.322295130" watchObservedRunningTime="2026-10-30 06:28:26.820477005 +0000 UTC m=+2073.329496894" Oct 30 06:28:27.146287 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:27.146249 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:27.575865 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:27.575792 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:29.292492 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:29.292451 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:31.231485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:31.231431 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:32.219531 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:32.219496 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:33.033160 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:33.033117 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:34.141817 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:28:34.141728 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:28:34.805344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:34.805305 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:36.146177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:36.146145 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:36.159658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:36.159630 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:36.599765 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:36.599680 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:36.853411 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:36.853337 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:28:38.351147 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:38.351106 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:39.612776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.612746 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn"] Oct 30 06:28:39.617985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.617965 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.652588 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.652511 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-6e2548\"/\"kube-root-ca.crt\"" Oct 30 06:28:39.652588 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.652527 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-6e2548\"/\"openshift-service-ca.crt\"" Oct 30 06:28:39.652846 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.652655 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-6e2548\"/\"default-dockercfg-hg6pd\"" Oct 30 06:28:39.652846 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.652775 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-6e2548\"/\"auth-test-model-6e2548-kserve-self-signed-certs\"" Oct 30 06:28:39.655614 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.655589 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn"] Oct 30 06:28:39.704286 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.704249 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd88e3b9-268f-4bc4-ad34-cd295e804e81-tls-certs\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.704507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.704343 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-dshm\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.704569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.704501 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-home\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.704679 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.704658 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dz6p\" (UniqueName: \"kubernetes.io/projected/dd88e3b9-268f-4bc4-ad34-cd295e804e81-kube-api-access-6dz6p\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.704737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.704710 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-tmp-dir\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.704792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.704757 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-model-cache\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.711326 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.711288 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:39.806454 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.806408 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dz6p\" (UniqueName: \"kubernetes.io/projected/dd88e3b9-268f-4bc4-ad34-cd295e804e81-kube-api-access-6dz6p\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.806656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.806468 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-tmp-dir\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.806656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.806512 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-model-cache\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.806782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.806678 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd88e3b9-268f-4bc4-ad34-cd295e804e81-tls-certs\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.806782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.806729 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-dshm\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.806782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.806769 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-home\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.806981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.806957 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-tmp-dir\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.807496 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.807467 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-model-cache\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.807692 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.807505 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-home\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.809621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.809587 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd88e3b9-268f-4bc4-ad34-cd295e804e81-tls-certs\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.809737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.809689 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-dshm\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.830758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.830702 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dz6p\" (UniqueName: \"kubernetes.io/projected/dd88e3b9-268f-4bc4-ad34-cd295e804e81-kube-api-access-6dz6p\") pod \"auth-test-model-6e2548-kserve-64478d5445-7njnn\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:39.932424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:39.932334 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:40.156472 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:40.156437 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:40.256653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:40.256626 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn"] Oct 30 06:28:40.257645 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:28:40.257619 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd88e3b9_268f_4bc4_ad34_cd295e804e81.slice/crio-f4309ad7b319270e5d58fceb746afa668b0a67ccc775596fdce5b53972abc9dc WatchSource:0}: Error finding container f4309ad7b319270e5d58fceb746afa668b0a67ccc775596fdce5b53972abc9dc: Status 404 returned error can't find the container with id f4309ad7b319270e5d58fceb746afa668b0a67ccc775596fdce5b53972abc9dc Oct 30 06:28:40.897483 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:40.897438 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" event={"ID":"dd88e3b9-268f-4bc4-ad34-cd295e804e81","Type":"ContainerStarted","Data":"05d16d2930743567812e9ea7cbff211b88bafdebd706d520c9fe60b19e8834be"} Oct 30 06:28:40.897939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:40.897511 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" event={"ID":"dd88e3b9-268f-4bc4-ad34-cd295e804e81","Type":"ContainerStarted","Data":"f4309ad7b319270e5d58fceb746afa668b0a67ccc775596fdce5b53972abc9dc"} Oct 30 06:28:40.933025 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:40.932992 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:40.942308 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:40.942256 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" podStartSLOduration=1.9422429559999999 podStartE2EDuration="1.942242956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:28:40.940290429 +0000 UTC m=+2087.449310317" watchObservedRunningTime="2026-10-30 06:28:40.942242956 +0000 UTC m=+2087.451262843" Oct 30 06:28:42.049481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:42.049446 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:43.866408 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:43.866377 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:45.792886 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:45.792841 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:46.719985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:46.719956 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:47.451994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:47.451967 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:48.806867 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:48.806837 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:49.478827 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:49.478795 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:49.932721 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:49.932686 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:49.945641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:49.945611 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:49.977296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:49.977270 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:28:51.510553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:51.510518 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:53.345493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:53.345460 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:55.139875 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:55.139819 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:56.932436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:56.932397 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:58.597294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:58.597263 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:58.756053 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:58.755988 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:28:59.014125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:59.014089 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq"] Oct 30 06:28:59.014393 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:28:59.014360 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" podUID="4fb98d9c-92c6-4aae-b631-f8319a76ee40" containerName="istio-proxy" containerID="cri-o://2ca9095be0fa03a7e0dfe51d8568f0de1ef5a69e08115dac0df393a593948b83" gracePeriod=30 Oct 30 06:29:00.496199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:00.496170 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:00.665123 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:00.665080 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:01.359842 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:01.359811 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:02.310851 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:02.310816 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:03.510020 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:03.509980 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:04.047831 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.047800 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:04.149863 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:29:04.149819 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:29:04.297584 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.297502 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:29:04.366626 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.366586 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb98d9c-92c6-4aae-b631-f8319a76ee40-kube-api-access-6hmqt\" (UniqueName: \"kubernetes.io/projected/4fb98d9c-92c6-4aae-b631-f8319a76ee40-kube-api-access-6hmqt\") pod \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " Oct 30 06:29:04.366830 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.366688 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-credential-socket\") pod \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " Oct 30 06:29:04.366830 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.366741 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-workload-certs\") pod \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " Oct 30 06:29:04.366830 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.366783 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-podinfo\") pod \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " Oct 30 06:29:04.367011 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.366872 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-token\" (UniqueName: \"kubernetes.io/projected/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-token\") pod \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " Oct 30 06:29:04.367011 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.366997 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-envoy\") pod \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " Oct 30 06:29:04.367129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.367062 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-credential-socket" pod "4fb98d9c-92c6-4aae-b631-f8319a76ee40" (UID: "4fb98d9c-92c6-4aae-b631-f8319a76ee40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:29:04.367129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.367117 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-workload-socket\") pod \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " Oct 30 06:29:04.367340 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.367194 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-data\") pod \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " Oct 30 06:29:04.367340 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.367242 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istiod-ca-cert\") pod \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\" (UID: \"4fb98d9c-92c6-4aae-b631-f8319a76ee40\") " Oct 30 06:29:04.367466 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.367412 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-workload-certs" pod "4fb98d9c-92c6-4aae-b631-f8319a76ee40" (UID: "4fb98d9c-92c6-4aae-b631-f8319a76ee40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:29:04.374008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.367943 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:29:04.374008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.367970 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:29:04.374008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.368051 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-workload-socket" pod "4fb98d9c-92c6-4aae-b631-f8319a76ee40" (UID: "4fb98d9c-92c6-4aae-b631-f8319a76ee40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:29:04.374008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.368177 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-data" pod "4fb98d9c-92c6-4aae-b631-f8319a76ee40" (UID: "4fb98d9c-92c6-4aae-b631-f8319a76ee40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:29:04.374008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.368274 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istiod-ca-cert" pod "4fb98d9c-92c6-4aae-b631-f8319a76ee40" (UID: "4fb98d9c-92c6-4aae-b631-f8319a76ee40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:29:04.374008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.371655 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-token" pod "4fb98d9c-92c6-4aae-b631-f8319a76ee40" (UID: "4fb98d9c-92c6-4aae-b631-f8319a76ee40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:29:04.374834 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.374791 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fb98d9c-92c6-4aae-b631-f8319a76ee40-kube-api-access-6hmqt" pod "4fb98d9c-92c6-4aae-b631-f8319a76ee40" (UID: "4fb98d9c-92c6-4aae-b631-f8319a76ee40"). InnerVolumeSpecName "kube-api-access-6hmqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:29:04.377399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.377364 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-podinfo" pod "4fb98d9c-92c6-4aae-b631-f8319a76ee40" (UID: "4fb98d9c-92c6-4aae-b631-f8319a76ee40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:29:04.378784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.378754 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-envoy" pod "4fb98d9c-92c6-4aae-b631-f8319a76ee40" (UID: "4fb98d9c-92c6-4aae-b631-f8319a76ee40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:29:04.469269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.469235 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:29:04.469269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.469270 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:29:04.469485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.469287 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:29:04.469485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.469302 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:29:04.469485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.469316 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hmqt\" (UniqueName: \"kubernetes.io/projected/4fb98d9c-92c6-4aae-b631-f8319a76ee40-kube-api-access-6hmqt\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:29:04.469485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.469328 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:29:04.469485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.469341 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fb98d9c-92c6-4aae-b631-f8319a76ee40-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:29:04.477035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.476998 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:04.715290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.715253 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph"] Oct 30 06:29:04.716683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.716637 6999 cpu_manager.go:414] "containerMap: removing container" podUID="4fb98d9c-92c6-4aae-b631-f8319a76ee40" containerName="istio-proxy" Oct 30 06:29:04.716683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.716680 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fb98d9c-92c6-4aae-b631-f8319a76ee40" containerName="istio-proxy" Oct 30 06:29:04.716883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.716833 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fb98d9c-92c6-4aae-b631-f8319a76ee40" containerName="istio-proxy" Oct 30 06:29:04.721207 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.721177 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.736481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.736454 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-3bbfc8\"/\"kube-root-ca.crt\"" Oct 30 06:29:04.736664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.736536 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-3bbfc8\"/\"openshift-service-ca.crt\"" Oct 30 06:29:04.736664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.736543 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-3bbfc8\"/\"explicit-model-3bbfc8-kserve-self-signed-certs\"" Oct 30 06:29:04.736664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.736629 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-3bbfc8\"/\"default-dockercfg-f9pq4\"" Oct 30 06:29:04.739163 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.739131 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph"] Oct 30 06:29:04.773807 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.773772 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-model-cache\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.774009 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.773821 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-dshm\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.774009 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.773911 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5hpz\" (UniqueName: \"kubernetes.io/projected/3822e7f9-25ce-4649-ae4d-b055884b6b42-kube-api-access-c5hpz\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.774147 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.774127 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3822e7f9-25ce-4649-ae4d-b055884b6b42-tls-certs\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.774187 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.774171 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-home\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.774272 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.774248 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-tmp-dir\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.799897 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.799858 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:04.876094 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.876055 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-tmp-dir\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.876317 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.876258 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-model-cache\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.876317 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.876291 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-dshm\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.876446 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.876322 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5hpz\" (UniqueName: \"kubernetes.io/projected/3822e7f9-25ce-4649-ae4d-b055884b6b42-kube-api-access-c5hpz\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.876515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.876472 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3822e7f9-25ce-4649-ae4d-b055884b6b42-tls-certs\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.876515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.876504 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-home\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.876627 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.876559 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-tmp-dir\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.876685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.876637 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-model-cache\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.876868 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.876839 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-home\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.879063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.879032 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-dshm\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.879297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.879281 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3822e7f9-25ce-4649-ae4d-b055884b6b42-tls-certs\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:04.900684 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:04.900644 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5hpz\" (UniqueName: \"kubernetes.io/projected/3822e7f9-25ce-4649-ae4d-b055884b6b42-kube-api-access-c5hpz\") pod \"explicit-model-3bbfc8-kserve-7c4597bfff-s2rph\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:05.037811 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:05.037639 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:05.081395 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:05.081356 6999 generic.go:356] "Generic (PLEG): container finished" podID="4fb98d9c-92c6-4aae-b631-f8319a76ee40" containerID="2ca9095be0fa03a7e0dfe51d8568f0de1ef5a69e08115dac0df393a593948b83" exitCode=0 Oct 30 06:29:05.081596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:05.081533 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" event={"ID":"4fb98d9c-92c6-4aae-b631-f8319a76ee40","Type":"ContainerDied","Data":"2ca9095be0fa03a7e0dfe51d8568f0de1ef5a69e08115dac0df393a593948b83"} Oct 30 06:29:05.081596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:05.081588 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" event={"ID":"4fb98d9c-92c6-4aae-b631-f8319a76ee40","Type":"ContainerDied","Data":"555542efd9b058aef1027775b3932b47455a1aa37ec7d4c20caa98e5fdc1f186"} Oct 30 06:29:05.081701 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:05.081618 6999 scope.go:122] "RemoveContainer" containerID="2ca9095be0fa03a7e0dfe51d8568f0de1ef5a69e08115dac0df393a593948b83" Oct 30 06:29:05.081815 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:05.081798 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq" Oct 30 06:29:05.106532 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:05.106504 6999 scope.go:122] "RemoveContainer" containerID="2ca9095be0fa03a7e0dfe51d8568f0de1ef5a69e08115dac0df393a593948b83" Oct 30 06:29:05.106931 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:29:05.106903 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca9095be0fa03a7e0dfe51d8568f0de1ef5a69e08115dac0df393a593948b83\": container with ID starting with 2ca9095be0fa03a7e0dfe51d8568f0de1ef5a69e08115dac0df393a593948b83 not found: ID does not exist" containerID="2ca9095be0fa03a7e0dfe51d8568f0de1ef5a69e08115dac0df393a593948b83" Oct 30 06:29:05.106996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:05.106947 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca9095be0fa03a7e0dfe51d8568f0de1ef5a69e08115dac0df393a593948b83"} err="failed to get container status \"2ca9095be0fa03a7e0dfe51d8568f0de1ef5a69e08115dac0df393a593948b83\": rpc error: code = NotFound desc = could not find container \"2ca9095be0fa03a7e0dfe51d8568f0de1ef5a69e08115dac0df393a593948b83\": container with ID starting with 2ca9095be0fa03a7e0dfe51d8568f0de1ef5a69e08115dac0df393a593948b83 not found: ID does not exist" Oct 30 06:29:05.156464 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:05.156428 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq"] Oct 30 06:29:05.163841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:05.163786 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-sp2jq"] Oct 30 06:29:05.382903 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:05.382871 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph"] Oct 30 06:29:05.384437 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:29:05.384410 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3822e7f9_25ce_4649_ae4d_b055884b6b42.slice/crio-532f33f5a81af7c5a1fce518b93220fc75704b25de777544bc8832a5eec1fb7a WatchSource:0}: Error finding container 532f33f5a81af7c5a1fce518b93220fc75704b25de777544bc8832a5eec1fb7a: Status 404 returned error can't find the container with id 532f33f5a81af7c5a1fce518b93220fc75704b25de777544bc8832a5eec1fb7a Oct 30 06:29:05.610534 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:05.610501 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fb98d9c-92c6-4aae-b631-f8319a76ee40" path="/var/lib/kubelet/pods/4fb98d9c-92c6-4aae-b631-f8319a76ee40/volumes" Oct 30 06:29:05.937142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:05.937107 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:06.090606 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:06.090573 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" event={"ID":"3822e7f9-25ce-4649-ae4d-b055884b6b42","Type":"ContainerStarted","Data":"c82cd489289df86f3fac453a24914a07d22f3ef2880b1532832df196967e2692"} Oct 30 06:29:06.090767 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:06.090616 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" event={"ID":"3822e7f9-25ce-4649-ae4d-b055884b6b42","Type":"ContainerStarted","Data":"532f33f5a81af7c5a1fce518b93220fc75704b25de777544bc8832a5eec1fb7a"} Oct 30 06:29:06.132835 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:06.132780 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" podStartSLOduration=2.132766521 podStartE2EDuration="2.132766521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:29:06.13144951 +0000 UTC m=+2112.640469397" watchObservedRunningTime="2026-10-30 06:29:06.132766521 +0000 UTC m=+2112.641786430" Oct 30 06:29:07.038292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:07.038203 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:07.741505 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:07.741468 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:09.502998 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:09.502962 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:11.226822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:11.226789 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:12.410951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:12.410920 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:13.019577 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:13.019545 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:14.742471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:14.742431 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:15.038100 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:15.038004 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:15.051051 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:15.051020 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:15.164535 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:15.164499 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:16.535332 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:16.535302 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:18.334299 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:18.334267 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:19.531592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.531536 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x"] Oct 30 06:29:19.536826 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.536798 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.569527 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.569491 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-868b1f\"/\"auth-test-model-868b1f-kserve-self-signed-certs\"" Oct 30 06:29:19.570308 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.569785 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-868b1f\"/\"kube-root-ca.crt\"" Oct 30 06:29:19.570483 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.569906 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-868b1f\"/\"openshift-service-ca.crt\"" Oct 30 06:29:19.570734 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.570716 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-868b1f\"/\"default-dockercfg-xdn7c\"" Oct 30 06:29:19.572396 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.572374 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x"] Oct 30 06:29:19.583624 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.583543 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-dshm\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.583624 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.583618 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-model-cache\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.583875 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.583683 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ead49a-073d-4bf3-bd25-afef28d6c83e-tls-certs\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.583875 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.583718 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-tmp-dir\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.583875 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.583785 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-home\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.583875 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.583815 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqz4g\" (UniqueName: \"kubernetes.io/projected/97ead49a-073d-4bf3-bd25-afef28d6c83e-kube-api-access-nqz4g\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.627400 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.627362 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:19.684800 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.684746 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-dshm\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.685010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.684825 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-model-cache\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.685010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.684851 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ead49a-073d-4bf3-bd25-afef28d6c83e-tls-certs\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.685010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.684879 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-tmp-dir\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.685010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.684973 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-home\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.685010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.684993 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqz4g\" (UniqueName: \"kubernetes.io/projected/97ead49a-073d-4bf3-bd25-afef28d6c83e-kube-api-access-nqz4g\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.685381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.685352 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-tmp-dir\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.685535 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.685416 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-home\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.685833 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.685806 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-model-cache\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.687798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.687769 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-dshm\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.688438 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.688417 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ead49a-073d-4bf3-bd25-afef28d6c83e-tls-certs\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.709383 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.709349 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqz4g\" (UniqueName: \"kubernetes.io/projected/97ead49a-073d-4bf3-bd25-afef28d6c83e-kube-api-access-nqz4g\") pod \"auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:19.853142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:19.853043 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:20.126367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:20.126331 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:20.169044 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:20.169023 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x"] Oct 30 06:29:20.170921 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:29:20.170896 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97ead49a_073d_4bf3_bd25_afef28d6c83e.slice/crio-116c12fc8cfc23a4adbf30df912c22c85a31b78e546d55db7f1ccb7ab480d3c6 WatchSource:0}: Error finding container 116c12fc8cfc23a4adbf30df912c22c85a31b78e546d55db7f1ccb7ab480d3c6: Status 404 returned error can't find the container with id 116c12fc8cfc23a4adbf30df912c22c85a31b78e546d55db7f1ccb7ab480d3c6 Oct 30 06:29:20.191955 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:20.191926 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" event={"ID":"97ead49a-073d-4bf3-bd25-afef28d6c83e","Type":"ContainerStarted","Data":"116c12fc8cfc23a4adbf30df912c22c85a31b78e546d55db7f1ccb7ab480d3c6"} Oct 30 06:29:21.200475 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:21.200433 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" event={"ID":"97ead49a-073d-4bf3-bd25-afef28d6c83e","Type":"ContainerStarted","Data":"59c85b859c1cc5e36e80551f436fdc2f6df021fedab07a7ebb60fcb3531b5c08"} Oct 30 06:29:21.242681 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:21.242633 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" podStartSLOduration=2.2426176509999998 podStartE2EDuration="2.242617651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:29:21.241453828 +0000 UTC m=+2127.750473717" watchObservedRunningTime="2026-10-30 06:29:21.242617651 +0000 UTC m=+2127.751637538" Oct 30 06:29:21.841431 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:21.841395 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:21.853804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:21.853768 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:23.625812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:23.625779 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:25.641888 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:25.641851 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:27.528061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:27.528024 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:27.696682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:27.696059 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph"] Oct 30 06:29:27.764113 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:27.764080 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:29.528673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:29.528638 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:29.854042 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:29.853946 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:29.866933 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:29.866904 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:30.273911 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:30.273875 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:29:31.362667 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:31.362633 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:33.146165 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:33.146096 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:34.144291 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:29:34.144252 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:29:34.943877 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:34.943839 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:36.731802 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:36.731762 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:38.536923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:38.536886 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:38.690073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:38.690038 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:40.279179 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:40.279144 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:42.056604 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:42.056575 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:42.727350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:42.727309 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" podUID="3822e7f9-25ce-4649-ae4d-b055884b6b42" containerName="main" containerID="cri-o://c82cd489289df86f3fac453a24914a07d22f3ef2880b1532832df196967e2692" gracePeriod=45 Oct 30 06:29:42.996078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:42.996018 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:43.040728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.039835 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-tmp-dir\") pod \"3822e7f9-25ce-4649-ae4d-b055884b6b42\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " Oct 30 06:29:43.040728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.039909 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-dshm\") pod \"3822e7f9-25ce-4649-ae4d-b055884b6b42\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " Oct 30 06:29:43.040728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.039983 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-model-cache\") pod \"3822e7f9-25ce-4649-ae4d-b055884b6b42\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " Oct 30 06:29:43.040728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.040055 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3822e7f9-25ce-4649-ae4d-b055884b6b42-kube-api-access-c5hpz\" (UniqueName: \"kubernetes.io/projected/3822e7f9-25ce-4649-ae4d-b055884b6b42-kube-api-access-c5hpz\") pod \"3822e7f9-25ce-4649-ae4d-b055884b6b42\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " Oct 30 06:29:43.040728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.040127 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-home\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-home\") pod \"3822e7f9-25ce-4649-ae4d-b055884b6b42\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " Oct 30 06:29:43.040728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.040331 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3822e7f9-25ce-4649-ae4d-b055884b6b42-tls-certs\" (UniqueName: \"kubernetes.io/secret/3822e7f9-25ce-4649-ae4d-b055884b6b42-tls-certs\") pod \"3822e7f9-25ce-4649-ae4d-b055884b6b42\" (UID: \"3822e7f9-25ce-4649-ae4d-b055884b6b42\") " Oct 30 06:29:43.041262 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.040981 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-home" pod "3822e7f9-25ce-4649-ae4d-b055884b6b42" (UID: "3822e7f9-25ce-4649-ae4d-b055884b6b42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:29:43.041262 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.040996 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-tmp-dir" pod "3822e7f9-25ce-4649-ae4d-b055884b6b42" (UID: "3822e7f9-25ce-4649-ae4d-b055884b6b42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:29:43.041581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.041547 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-model-cache" pod "3822e7f9-25ce-4649-ae4d-b055884b6b42" (UID: "3822e7f9-25ce-4649-ae4d-b055884b6b42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:29:43.041694 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.041674 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:29:43.041694 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.041696 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:29:43.041860 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.041708 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:29:43.043784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.043760 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3822e7f9-25ce-4649-ae4d-b055884b6b42-kube-api-access-c5hpz" pod "3822e7f9-25ce-4649-ae4d-b055884b6b42" (UID: "3822e7f9-25ce-4649-ae4d-b055884b6b42"). InnerVolumeSpecName "kube-api-access-c5hpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:29:43.044351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.044329 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3822e7f9-25ce-4649-ae4d-b055884b6b42-tls-certs" pod "3822e7f9-25ce-4649-ae4d-b055884b6b42" (UID: "3822e7f9-25ce-4649-ae4d-b055884b6b42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:29:43.045337 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.045314 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-dshm" pod "3822e7f9-25ce-4649-ae4d-b055884b6b42" (UID: "3822e7f9-25ce-4649-ae4d-b055884b6b42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:29:43.142622 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.142596 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3822e7f9-25ce-4649-ae4d-b055884b6b42-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:29:43.142622 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.142619 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3822e7f9-25ce-4649-ae4d-b055884b6b42-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:29:43.143010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.142627 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5hpz\" (UniqueName: \"kubernetes.io/projected/3822e7f9-25ce-4649-ae4d-b055884b6b42-kube-api-access-c5hpz\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:29:43.359539 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.359504 6999 generic.go:356] "Generic (PLEG): container finished" podID="3822e7f9-25ce-4649-ae4d-b055884b6b42" containerID="c82cd489289df86f3fac453a24914a07d22f3ef2880b1532832df196967e2692" exitCode=0 Oct 30 06:29:43.359691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.359555 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" Oct 30 06:29:43.359691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.359584 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" event={"ID":"3822e7f9-25ce-4649-ae4d-b055884b6b42","Type":"ContainerDied","Data":"c82cd489289df86f3fac453a24914a07d22f3ef2880b1532832df196967e2692"} Oct 30 06:29:43.359691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.359627 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph" event={"ID":"3822e7f9-25ce-4649-ae4d-b055884b6b42","Type":"ContainerDied","Data":"532f33f5a81af7c5a1fce518b93220fc75704b25de777544bc8832a5eec1fb7a"} Oct 30 06:29:43.359691 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.359643 6999 scope.go:122] "RemoveContainer" containerID="c82cd489289df86f3fac453a24914a07d22f3ef2880b1532832df196967e2692" Oct 30 06:29:43.370113 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.370091 6999 scope.go:122] "RemoveContainer" containerID="c82cd489289df86f3fac453a24914a07d22f3ef2880b1532832df196967e2692" Oct 30 06:29:43.370399 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:29:43.370378 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c82cd489289df86f3fac453a24914a07d22f3ef2880b1532832df196967e2692\": container with ID starting with c82cd489289df86f3fac453a24914a07d22f3ef2880b1532832df196967e2692 not found: ID does not exist" containerID="c82cd489289df86f3fac453a24914a07d22f3ef2880b1532832df196967e2692" Oct 30 06:29:43.370487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.370412 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c82cd489289df86f3fac453a24914a07d22f3ef2880b1532832df196967e2692"} err="failed to get container status \"c82cd489289df86f3fac453a24914a07d22f3ef2880b1532832df196967e2692\": rpc error: code = NotFound desc = could not find container \"c82cd489289df86f3fac453a24914a07d22f3ef2880b1532832df196967e2692\": container with ID starting with c82cd489289df86f3fac453a24914a07d22f3ef2880b1532832df196967e2692 not found: ID does not exist" Oct 30 06:29:43.411886 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.411852 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph"] Oct 30 06:29:43.414723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.414695 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-3bbfc8/explicit-model-3bbfc8-kserve-7c4597bfff-s2rph"] Oct 30 06:29:43.610986 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.610904 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3822e7f9-25ce-4649-ae4d-b055884b6b42" path="/var/lib/kubelet/pods/3822e7f9-25ce-4649-ae4d-b055884b6b42/volumes" Oct 30 06:29:43.757773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:43.757741 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:45.580530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:45.580491 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:47.322203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:47.322169 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:49.007180 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:49.007063 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:50.844466 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:50.844433 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:52.622618 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:52.622583 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:53.774789 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:53.774751 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:54.421325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:54.421286 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:56.355163 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:56.355124 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:29:58.233319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:29:58.233274 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:00.000140 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.000097 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:00.209418 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.209386 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s"] Oct 30 06:30:00.210474 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.210422 6999 cpu_manager.go:414] "containerMap: removing container" podUID="3822e7f9-25ce-4649-ae4d-b055884b6b42" containerName="main" Oct 30 06:30:00.210474 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.210446 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3822e7f9-25ce-4649-ae4d-b055884b6b42" containerName="main" Oct 30 06:30:00.210692 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.210608 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="3822e7f9-25ce-4649-ae4d-b055884b6b42" containerName="main" Oct 30 06:30:00.216207 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.216184 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" Oct 30 06:30:00.237075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.237039 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s"] Oct 30 06:30:00.274128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.274050 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 06:30:00.279644 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.279613 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98l7n\" (UniqueName: \"kubernetes.io/projected/e4230847-8bb3-48c1-99b6-550fa76c4b10-kube-api-access-98l7n\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s\" (UID: \"e4230847-8bb3-48c1-99b6-550fa76c4b10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" Oct 30 06:30:00.279823 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.279799 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4230847-8bb3-48c1-99b6-550fa76c4b10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s\" (UID: \"e4230847-8bb3-48c1-99b6-550fa76c4b10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" Oct 30 06:30:00.380926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.380893 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4230847-8bb3-48c1-99b6-550fa76c4b10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s\" (UID: \"e4230847-8bb3-48c1-99b6-550fa76c4b10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" Oct 30 06:30:00.381117 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.381053 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98l7n\" (UniqueName: \"kubernetes.io/projected/e4230847-8bb3-48c1-99b6-550fa76c4b10-kube-api-access-98l7n\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s\" (UID: \"e4230847-8bb3-48c1-99b6-550fa76c4b10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" Oct 30 06:30:00.381625 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.381605 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4230847-8bb3-48c1-99b6-550fa76c4b10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s\" (UID: \"e4230847-8bb3-48c1-99b6-550fa76c4b10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" Oct 30 06:30:00.404760 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.404732 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98l7n\" (UniqueName: \"kubernetes.io/projected/e4230847-8bb3-48c1-99b6-550fa76c4b10-kube-api-access-98l7n\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s\" (UID: \"e4230847-8bb3-48c1-99b6-550fa76c4b10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" Oct 30 06:30:00.540541 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.540457 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" Oct 30 06:30:00.844640 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:00.844608 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s"] Oct 30 06:30:00.846229 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:30:00.846190 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4230847_8bb3_48c1_99b6_550fa76c4b10.slice/crio-38765ea4aae02f80741056fa1bf621377da9e9672663dbbddfe0f13c62cf048a WatchSource:0}: Error finding container 38765ea4aae02f80741056fa1bf621377da9e9672663dbbddfe0f13c62cf048a: Status 404 returned error can't find the container with id 38765ea4aae02f80741056fa1bf621377da9e9672663dbbddfe0f13c62cf048a Oct 30 06:30:01.477863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:01.477771 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" event={"ID":"e4230847-8bb3-48c1-99b6-550fa76c4b10","Type":"ContainerStarted","Data":"74aa6711a7ce4e935473cb446bab8b303f28ae1935ee44fb8a3c75f45abcdadd"} Oct 30 06:30:01.477863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:01.477821 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" event={"ID":"e4230847-8bb3-48c1-99b6-550fa76c4b10","Type":"ContainerStarted","Data":"38765ea4aae02f80741056fa1bf621377da9e9672663dbbddfe0f13c62cf048a"} Oct 30 06:30:01.518424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:01.518372 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" podStartSLOduration=1.51835852 podStartE2EDuration="1.51835852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:01.517045512 +0000 UTC m=+2168.026065400" watchObservedRunningTime="2026-10-30 06:30:01.51835852 +0000 UTC m=+2168.027378406" Oct 30 06:30:01.671547 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:01.671512 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:01.819760 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:01.819721 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:02.487350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:02.487317 6999 generic.go:356] "Generic (PLEG): container finished" podID="e4230847-8bb3-48c1-99b6-550fa76c4b10" containerID="74aa6711a7ce4e935473cb446bab8b303f28ae1935ee44fb8a3c75f45abcdadd" exitCode=60 Oct 30 06:30:02.487910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:02.487409 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" event={"ID":"e4230847-8bb3-48c1-99b6-550fa76c4b10","Type":"ContainerDied","Data":"74aa6711a7ce4e935473cb446bab8b303f28ae1935ee44fb8a3c75f45abcdadd"} Oct 30 06:30:02.487910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:02.487701 6999 scope.go:122] "RemoveContainer" containerID="74aa6711a7ce4e935473cb446bab8b303f28ae1935ee44fb8a3c75f45abcdadd" Oct 30 06:30:03.249739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.249702 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg"] Oct 30 06:30:03.249949 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.249917 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" podUID="e0aee967-261a-42ce-9e34-2d62082786ed" containerName="maas-api" containerID="cri-o://d092ac43acb07cdd7b9a95b6dd6f06b2d26e50333ac23227df00bbc02423cc03" gracePeriod=30 Oct 30 06:30:03.355888 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.355851 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:03.542239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.539465 6999 generic.go:356] "Generic (PLEG): container finished" podID="e4230847-8bb3-48c1-99b6-550fa76c4b10" containerID="447eb0c460e95bbc32a0d4f2c6e995cab1980e4bc1248a25d123cd9a4b2d8e40" exitCode=60 Oct 30 06:30:03.542239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.539699 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" event={"ID":"e4230847-8bb3-48c1-99b6-550fa76c4b10","Type":"ContainerDied","Data":"447eb0c460e95bbc32a0d4f2c6e995cab1980e4bc1248a25d123cd9a4b2d8e40"} Oct 30 06:30:03.542239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.539761 6999 scope.go:122] "RemoveContainer" containerID="74aa6711a7ce4e935473cb446bab8b303f28ae1935ee44fb8a3c75f45abcdadd" Oct 30 06:30:03.544146 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.544115 6999 scope.go:122] "RemoveContainer" containerID="447eb0c460e95bbc32a0d4f2c6e995cab1980e4bc1248a25d123cd9a4b2d8e40" Oct 30 06:30:03.544628 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:30:03.544597 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s_odh-ai-gateway-infra(e4230847-8bb3-48c1-99b6-550fa76c4b10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" podUID="e4230847-8bb3-48c1-99b6-550fa76c4b10" Oct 30 06:30:03.560248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.550259 6999 generic.go:356] "Generic (PLEG): container finished" podID="e0aee967-261a-42ce-9e34-2d62082786ed" containerID="d092ac43acb07cdd7b9a95b6dd6f06b2d26e50333ac23227df00bbc02423cc03" exitCode=0 Oct 30 06:30:03.560248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.550512 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" event={"ID":"e0aee967-261a-42ce-9e34-2d62082786ed","Type":"ContainerDied","Data":"d092ac43acb07cdd7b9a95b6dd6f06b2d26e50333ac23227df00bbc02423cc03"} Oct 30 06:30:03.623084 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.623049 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:30:03.636961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.636923 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0aee967-261a-42ce-9e34-2d62082786ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0aee967-261a-42ce-9e34-2d62082786ed-openshift-service-ca\") pod \"e0aee967-261a-42ce-9e34-2d62082786ed\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " Oct 30 06:30:03.637157 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.637030 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0aee967-261a-42ce-9e34-2d62082786ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0aee967-261a-42ce-9e34-2d62082786ed-metrics-certs\") pod \"e0aee967-261a-42ce-9e34-2d62082786ed\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " Oct 30 06:30:03.637250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.637178 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0aee967-261a-42ce-9e34-2d62082786ed-kube-api-access-4s724\" (UniqueName: \"kubernetes.io/projected/e0aee967-261a-42ce-9e34-2d62082786ed-kube-api-access-4s724\") pod \"e0aee967-261a-42ce-9e34-2d62082786ed\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " Oct 30 06:30:03.637369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.637352 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0aee967-261a-42ce-9e34-2d62082786ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0aee967-261a-42ce-9e34-2d62082786ed-maas-api-tls\") pod \"e0aee967-261a-42ce-9e34-2d62082786ed\" (UID: \"e0aee967-261a-42ce-9e34-2d62082786ed\") " Oct 30 06:30:03.637488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.637351 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0aee967-261a-42ce-9e34-2d62082786ed-openshift-service-ca" pod "e0aee967-261a-42ce-9e34-2d62082786ed" (UID: "e0aee967-261a-42ce-9e34-2d62082786ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:03.638174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.637967 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0aee967-261a-42ce-9e34-2d62082786ed-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:30:03.640599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.640565 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0aee967-261a-42ce-9e34-2d62082786ed-maas-api-tls" pod "e0aee967-261a-42ce-9e34-2d62082786ed" (UID: "e0aee967-261a-42ce-9e34-2d62082786ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:30:03.640725 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.640643 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0aee967-261a-42ce-9e34-2d62082786ed-metrics-certs" pod "e0aee967-261a-42ce-9e34-2d62082786ed" (UID: "e0aee967-261a-42ce-9e34-2d62082786ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:30:03.640787 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.640758 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0aee967-261a-42ce-9e34-2d62082786ed-kube-api-access-4s724" pod "e0aee967-261a-42ce-9e34-2d62082786ed" (UID: "e0aee967-261a-42ce-9e34-2d62082786ed"). InnerVolumeSpecName "kube-api-access-4s724". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:03.738942 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.738835 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0aee967-261a-42ce-9e34-2d62082786ed-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:30:03.738942 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.738872 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0aee967-261a-42ce-9e34-2d62082786ed-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:30:03.738942 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.738888 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s724\" (UniqueName: \"kubernetes.io/projected/e0aee967-261a-42ce-9e34-2d62082786ed-kube-api-access-4s724\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:30:03.914869 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:03.914838 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:04.056982 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:04.056945 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:04.148280 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:30:04.148186 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:30:04.202534 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:04.202447 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:04.559139 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:04.559053 6999 scope.go:122] "RemoveContainer" containerID="447eb0c460e95bbc32a0d4f2c6e995cab1980e4bc1248a25d123cd9a4b2d8e40" Oct 30 06:30:04.559644 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:30:04.559407 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s_odh-ai-gateway-infra(e4230847-8bb3-48c1-99b6-550fa76c4b10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" podUID="e4230847-8bb3-48c1-99b6-550fa76c4b10" Oct 30 06:30:04.560246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:04.560197 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" Oct 30 06:30:04.560371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:04.560257 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg" event={"ID":"e0aee967-261a-42ce-9e34-2d62082786ed","Type":"ContainerDied","Data":"c34c57f9fb65e7d95d8a37113155b6ed80e3175eea782535566e754b113ce73d"} Oct 30 06:30:04.560371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:04.560325 6999 scope.go:122] "RemoveContainer" containerID="d092ac43acb07cdd7b9a95b6dd6f06b2d26e50333ac23227df00bbc02423cc03" Oct 30 06:30:04.627047 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:04.627020 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:04.653360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:04.653327 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg"] Oct 30 06:30:04.658162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:04.658135 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-3bbfc8-df989896-v94cg"] Oct 30 06:30:05.614199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:05.614168 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0aee967-261a-42ce-9e34-2d62082786ed" path="/var/lib/kubelet/pods/e0aee967-261a-42ce-9e34-2d62082786ed/volumes" Oct 30 06:30:06.083884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:06.083804 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:07.943120 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:07.943089 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:09.743632 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:09.743560 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:11.202675 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:11.202644 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:11.349453 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:11.349420 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:11.514353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:11.514257 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:11.666342 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:11.666293 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:12.255645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:12.255602 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:12.747192 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:12.747164 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:12.916753 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:12.916723 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:13.326335 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:13.326307 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:14.902503 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:14.902467 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:15.041872 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:15.041840 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7"] Oct 30 06:30:15.049835 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:15.049811 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7f5c011-tsmn7"] Oct 30 06:30:15.057901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:15.057877 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:15.204975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:15.204896 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:15.357391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:15.357359 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:15.612667 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:15.612638 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3605490-9f6d-4acd-9cba-631cd9d017c2" path="/var/lib/kubelet/pods/a3605490-9f6d-4acd-9cba-631cd9d017c2/volumes" Oct 30 06:30:16.702903 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.702869 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht"] Oct 30 06:30:16.703978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.703954 6999 cpu_manager.go:414] "containerMap: removing container" podUID="e0aee967-261a-42ce-9e34-2d62082786ed" containerName="maas-api" Oct 30 06:30:16.703978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.703977 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0aee967-261a-42ce-9e34-2d62082786ed" containerName="maas-api" Oct 30 06:30:16.704163 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.704148 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0aee967-261a-42ce-9e34-2d62082786ed" containerName="maas-api" Oct 30 06:30:16.708114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.708090 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.736509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.736479 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-b8b590\"/\"kube-root-ca.crt\"" Oct 30 06:30:16.736509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.736479 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-b8b590\"/\"openshift-service-ca.crt\"" Oct 30 06:30:16.736509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.736479 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-b8b590\"/\"noten-model-b8b590-kserve-self-signed-certs\"" Oct 30 06:30:16.736788 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.736543 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-b8b590\"/\"default-dockercfg-6lqwz\"" Oct 30 06:30:16.741141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.741112 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht"] Oct 30 06:30:16.792816 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.792785 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:16.828683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.828651 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124569c5-b186-461a-8d4c-6f32c13559aa-tls-certs\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.828872 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.828695 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-model-cache\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.828872 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.828717 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-home\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.828872 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.828740 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-dshm\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.828872 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.828855 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9jdp\" (UniqueName: \"kubernetes.io/projected/124569c5-b186-461a-8d4c-6f32c13559aa-kube-api-access-x9jdp\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.829074 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.829030 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-tmp-dir\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.931233 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.930594 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9jdp\" (UniqueName: \"kubernetes.io/projected/124569c5-b186-461a-8d4c-6f32c13559aa-kube-api-access-x9jdp\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.931233 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.930773 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-tmp-dir\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.931233 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.930956 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124569c5-b186-461a-8d4c-6f32c13559aa-tls-certs\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.931233 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.930999 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-model-cache\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.931233 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.931032 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-home\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.931233 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.931066 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-dshm\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.931687 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.931458 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-tmp-dir\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.932151 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.932040 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-model-cache\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.932573 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.932549 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-home\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.935833 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.935791 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124569c5-b186-461a-8d4c-6f32c13559aa-tls-certs\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.936420 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.936383 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-dshm\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:16.964155 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:16.964044 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9jdp\" (UniqueName: \"kubernetes.io/projected/124569c5-b186-461a-8d4c-6f32c13559aa-kube-api-access-x9jdp\") pod \"noten-model-b8b590-kserve-565776fb4f-8r4ht\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:17.028670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:17.027842 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:17.174249 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:17.172401 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:17.359818 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:17.359790 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht"] Oct 30 06:30:17.362080 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:30:17.362055 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod124569c5_b186_461a_8d4c_6f32c13559aa.slice/crio-2c7da24b1f6d3712c670fbfa6e32bad3899e2e073cac8b736ff23a28a36c0ef0 WatchSource:0}: Error finding container 2c7da24b1f6d3712c670fbfa6e32bad3899e2e073cac8b736ff23a28a36c0ef0: Status 404 returned error can't find the container with id 2c7da24b1f6d3712c670fbfa6e32bad3899e2e073cac8b736ff23a28a36c0ef0 Oct 30 06:30:17.601677 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:17.601644 6999 scope.go:122] "RemoveContainer" containerID="447eb0c460e95bbc32a0d4f2c6e995cab1980e4bc1248a25d123cd9a4b2d8e40" Oct 30 06:30:17.655883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:17.655843 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" event={"ID":"124569c5-b186-461a-8d4c-6f32c13559aa","Type":"ContainerStarted","Data":"e07b31afdf07e8422e513438c67feea5e7eaeb68fad9916835d2ce5c5e7b3332"} Oct 30 06:30:17.656088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:17.655896 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" event={"ID":"124569c5-b186-461a-8d4c-6f32c13559aa","Type":"ContainerStarted","Data":"2c7da24b1f6d3712c670fbfa6e32bad3899e2e073cac8b736ff23a28a36c0ef0"} Oct 30 06:30:18.029353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:18.029326 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:18.668062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:18.668032 6999 generic.go:356] "Generic (PLEG): container finished" podID="e4230847-8bb3-48c1-99b6-550fa76c4b10" containerID="77ec711f7f76cb3f769dc3691b9b9f9465bc2453809dcb2a5c91d7bffeab58c6" exitCode=60 Oct 30 06:30:18.668288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:18.668118 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" event={"ID":"e4230847-8bb3-48c1-99b6-550fa76c4b10","Type":"ContainerDied","Data":"77ec711f7f76cb3f769dc3691b9b9f9465bc2453809dcb2a5c91d7bffeab58c6"} Oct 30 06:30:18.668288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:18.668197 6999 scope.go:122] "RemoveContainer" containerID="447eb0c460e95bbc32a0d4f2c6e995cab1980e4bc1248a25d123cd9a4b2d8e40" Oct 30 06:30:18.668932 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:18.668896 6999 scope.go:122] "RemoveContainer" containerID="77ec711f7f76cb3f769dc3691b9b9f9465bc2453809dcb2a5c91d7bffeab58c6" Oct 30 06:30:18.669268 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:30:18.669201 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s_odh-ai-gateway-infra(e4230847-8bb3-48c1-99b6-550fa76c4b10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" podUID="e4230847-8bb3-48c1-99b6-550fa76c4b10" Oct 30 06:30:18.712510 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:18.712463 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" podStartSLOduration=2.712449481 podStartE2EDuration="2.712449481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:17.700714746 +0000 UTC m=+2184.209734634" watchObservedRunningTime="2026-10-30 06:30:18.712449481 +0000 UTC m=+2185.221469367" Oct 30 06:30:19.051836 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:19.051793 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:19.723477 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:19.723450 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s"] Oct 30 06:30:19.884541 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:19.884513 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" Oct 30 06:30:19.977697 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:19.977613 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4230847-8bb3-48c1-99b6-550fa76c4b10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4230847-8bb3-48c1-99b6-550fa76c4b10-maas-api-service-ca\") pod \"e4230847-8bb3-48c1-99b6-550fa76c4b10\" (UID: \"e4230847-8bb3-48c1-99b6-550fa76c4b10\") " Oct 30 06:30:19.977697 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:19.977670 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4230847-8bb3-48c1-99b6-550fa76c4b10-kube-api-access-98l7n\" (UniqueName: \"kubernetes.io/projected/e4230847-8bb3-48c1-99b6-550fa76c4b10-kube-api-access-98l7n\") pod \"e4230847-8bb3-48c1-99b6-550fa76c4b10\" (UID: \"e4230847-8bb3-48c1-99b6-550fa76c4b10\") " Oct 30 06:30:19.978008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:19.977987 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4230847-8bb3-48c1-99b6-550fa76c4b10-maas-api-service-ca" pod "e4230847-8bb3-48c1-99b6-550fa76c4b10" (UID: "e4230847-8bb3-48c1-99b6-550fa76c4b10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:19.978492 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:19.978329 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4230847-8bb3-48c1-99b6-550fa76c4b10-maas-api-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:30:19.985707 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:19.985676 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4230847-8bb3-48c1-99b6-550fa76c4b10-kube-api-access-98l7n" pod "e4230847-8bb3-48c1-99b6-550fa76c4b10" (UID: "e4230847-8bb3-48c1-99b6-550fa76c4b10"). InnerVolumeSpecName "kube-api-access-98l7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:20.079667 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:20.079634 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98l7n\" (UniqueName: \"kubernetes.io/projected/e4230847-8bb3-48c1-99b6-550fa76c4b10-kube-api-access-98l7n\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:30:20.686793 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:20.686756 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" event={"ID":"e4230847-8bb3-48c1-99b6-550fa76c4b10","Type":"ContainerDied","Data":"38765ea4aae02f80741056fa1bf621377da9e9672663dbbddfe0f13c62cf048a"} Oct 30 06:30:20.686793 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:20.686790 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s" Oct 30 06:30:20.687055 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:20.686818 6999 scope.go:122] "RemoveContainer" containerID="77ec711f7f76cb3f769dc3691b9b9f9465bc2453809dcb2a5c91d7bffeab58c6" Oct 30 06:30:20.739474 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:20.739442 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s"] Oct 30 06:30:20.751753 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:20.751722 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-adcdcb-29889030-z5r4s"] Oct 30 06:30:21.060653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:21.060555 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:21.609994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:21.609961 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4230847-8bb3-48c1-99b6-550fa76c4b10" path="/var/lib/kubelet/pods/e4230847-8bb3-48c1-99b6-550fa76c4b10/volumes" Oct 30 06:30:22.836634 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:22.836598 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:24.642270 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:24.642238 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:26.424995 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:26.424963 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:27.029687 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:27.029653 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:27.042883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:27.042852 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:27.745685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:27.745656 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:28.272778 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:28.272745 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:30.129199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:30.129166 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:31.915501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:31.915471 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:33.701716 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:33.701684 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:34.143150 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:30:34.143099 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:30:35.415080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:35.414999 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:37.130906 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:37.130873 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:38.919064 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:38.919030 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:40.196120 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:40.196080 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht"] Oct 30 06:30:40.355745 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:40.355711 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:40.531343 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:40.531208 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:40.821592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:40.821510 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:40.964674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:40.964628 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:42.749703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:42.749671 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:43.942141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:43.942105 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:44.439767 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:44.439741 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:45.997524 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:45.997491 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:46.231321 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:46.231285 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:48.012969 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:48.012936 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:49.642510 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:49.642474 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:49.847339 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:49.847304 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:51.603844 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:51.603810 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:53.383092 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:53.383051 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:55.135860 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.135794 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:55.220629 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.220583 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" podUID="124569c5-b186-461a-8d4c-6f32c13559aa" containerName="main" containerID="cri-o://e07b31afdf07e8422e513438c67feea5e7eaeb68fad9916835d2ce5c5e7b3332" gracePeriod=45 Oct 30 06:30:55.506538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.506512 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:55.688598 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.688514 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-model-cache\") pod \"124569c5-b186-461a-8d4c-6f32c13559aa\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " Oct 30 06:30:55.688768 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.688655 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/124569c5-b186-461a-8d4c-6f32c13559aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/124569c5-b186-461a-8d4c-6f32c13559aa-tls-certs\") pod \"124569c5-b186-461a-8d4c-6f32c13559aa\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " Oct 30 06:30:55.688768 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.688713 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-dshm\") pod \"124569c5-b186-461a-8d4c-6f32c13559aa\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " Oct 30 06:30:55.688913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.688825 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124569c5-b186-461a-8d4c-6f32c13559aa-kube-api-access-x9jdp\" (UniqueName: \"kubernetes.io/projected/124569c5-b186-461a-8d4c-6f32c13559aa-kube-api-access-x9jdp\") pod \"124569c5-b186-461a-8d4c-6f32c13559aa\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " Oct 30 06:30:55.688913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.688850 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-model-cache" pod "124569c5-b186-461a-8d4c-6f32c13559aa" (UID: "124569c5-b186-461a-8d4c-6f32c13559aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:30:55.688913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.688900 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-tmp-dir\") pod \"124569c5-b186-461a-8d4c-6f32c13559aa\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " Oct 30 06:30:55.689076 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.688932 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-home\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-home\") pod \"124569c5-b186-461a-8d4c-6f32c13559aa\" (UID: \"124569c5-b186-461a-8d4c-6f32c13559aa\") " Oct 30 06:30:55.689168 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.689149 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-tmp-dir" pod "124569c5-b186-461a-8d4c-6f32c13559aa" (UID: "124569c5-b186-461a-8d4c-6f32c13559aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:30:55.689430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.689378 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-home" pod "124569c5-b186-461a-8d4c-6f32c13559aa" (UID: "124569c5-b186-461a-8d4c-6f32c13559aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:30:55.689710 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.689690 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:30:55.689794 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.689715 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:30:55.689794 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.689730 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:30:55.691181 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.691153 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124569c5-b186-461a-8d4c-6f32c13559aa-kube-api-access-x9jdp" pod "124569c5-b186-461a-8d4c-6f32c13559aa" (UID: "124569c5-b186-461a-8d4c-6f32c13559aa"). InnerVolumeSpecName "kube-api-access-x9jdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:55.691333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.691180 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-dshm" pod "124569c5-b186-461a-8d4c-6f32c13559aa" (UID: "124569c5-b186-461a-8d4c-6f32c13559aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:30:55.691333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.691275 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/124569c5-b186-461a-8d4c-6f32c13559aa-tls-certs" pod "124569c5-b186-461a-8d4c-6f32c13559aa" (UID: "124569c5-b186-461a-8d4c-6f32c13559aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:30:55.790598 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.790572 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124569c5-b186-461a-8d4c-6f32c13559aa-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:30:55.790598 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.790596 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124569c5-b186-461a-8d4c-6f32c13559aa-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:30:55.790774 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.790609 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9jdp\" (UniqueName: \"kubernetes.io/projected/124569c5-b186-461a-8d4c-6f32c13559aa-kube-api-access-x9jdp\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:30:55.925292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.925248 6999 generic.go:356] "Generic (PLEG): container finished" podID="124569c5-b186-461a-8d4c-6f32c13559aa" containerID="e07b31afdf07e8422e513438c67feea5e7eaeb68fad9916835d2ce5c5e7b3332" exitCode=0 Oct 30 06:30:55.925508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.925453 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" Oct 30 06:30:55.925508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.925491 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" event={"ID":"124569c5-b186-461a-8d4c-6f32c13559aa","Type":"ContainerDied","Data":"e07b31afdf07e8422e513438c67feea5e7eaeb68fad9916835d2ce5c5e7b3332"} Oct 30 06:30:55.925645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.925534 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht" event={"ID":"124569c5-b186-461a-8d4c-6f32c13559aa","Type":"ContainerDied","Data":"2c7da24b1f6d3712c670fbfa6e32bad3899e2e073cac8b736ff23a28a36c0ef0"} Oct 30 06:30:55.925645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.925552 6999 scope.go:122] "RemoveContainer" containerID="e07b31afdf07e8422e513438c67feea5e7eaeb68fad9916835d2ce5c5e7b3332" Oct 30 06:30:55.936021 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.936005 6999 scope.go:122] "RemoveContainer" containerID="e07b31afdf07e8422e513438c67feea5e7eaeb68fad9916835d2ce5c5e7b3332" Oct 30 06:30:55.936284 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:30:55.936265 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07b31afdf07e8422e513438c67feea5e7eaeb68fad9916835d2ce5c5e7b3332\": container with ID starting with e07b31afdf07e8422e513438c67feea5e7eaeb68fad9916835d2ce5c5e7b3332 not found: ID does not exist" containerID="e07b31afdf07e8422e513438c67feea5e7eaeb68fad9916835d2ce5c5e7b3332" Oct 30 06:30:55.936345 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.936293 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07b31afdf07e8422e513438c67feea5e7eaeb68fad9916835d2ce5c5e7b3332"} err="failed to get container status \"e07b31afdf07e8422e513438c67feea5e7eaeb68fad9916835d2ce5c5e7b3332\": rpc error: code = NotFound desc = could not find container \"e07b31afdf07e8422e513438c67feea5e7eaeb68fad9916835d2ce5c5e7b3332\": container with ID starting with e07b31afdf07e8422e513438c67feea5e7eaeb68fad9916835d2ce5c5e7b3332 not found: ID does not exist" Oct 30 06:30:55.978016 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.977961 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht"] Oct 30 06:30:55.980392 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:55.980369 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-b8b590/noten-model-b8b590-kserve-565776fb4f-8r4ht"] Oct 30 06:30:56.892383 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:56.892329 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:30:57.610548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:57.610514 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="124569c5-b186-461a-8d4c-6f32c13559aa" path="/var/lib/kubelet/pods/124569c5-b186-461a-8d4c-6f32c13559aa/volumes" Oct 30 06:30:58.583676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:30:58.583633 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:00.330288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:00.330259 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:01.875250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:01.875202 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:02.043799 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:02.043768 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:02.280207 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:02.280130 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:03.776018 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:03.775990 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:04.142338 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:04.142299 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:31:04.261794 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:04.261759 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:05.630536 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:05.630499 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:07.342155 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:07.342119 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:09.042898 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:09.042868 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:10.875045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:10.874991 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:11.594050 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:11.594016 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz"] Oct 30 06:31:11.664652 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:11.664619 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:12.627501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:12.627465 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:14.340186 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:14.340158 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:16.137771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:16.137732 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:18.024748 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:18.024714 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:19.743529 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:19.743474 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:21.506929 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:21.506897 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:21.928711 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:21.928675 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95"] Oct 30 06:31:22.134114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.134078 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:22.271861 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.271782 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7315-c172b3"] Oct 30 06:31:22.272408 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.272391 6999 cpu_manager.go:414] "containerMap: removing container" podUID="e4230847-8bb3-48c1-99b6-550fa76c4b10" containerName="cleanup" Oct 30 06:31:22.272408 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.272409 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4230847-8bb3-48c1-99b6-550fa76c4b10" containerName="cleanup" Oct 30 06:31:22.272583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.272417 6999 cpu_manager.go:414] "containerMap: removing container" podUID="e4230847-8bb3-48c1-99b6-550fa76c4b10" containerName="cleanup" Oct 30 06:31:22.272583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.272422 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4230847-8bb3-48c1-99b6-550fa76c4b10" containerName="cleanup" Oct 30 06:31:22.272583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.272454 6999 cpu_manager.go:414] "containerMap: removing container" podUID="124569c5-b186-461a-8d4c-6f32c13559aa" containerName="main" Oct 30 06:31:22.272583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.272464 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124569c5-b186-461a-8d4c-6f32c13559aa" containerName="main" Oct 30 06:31:22.272720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.272616 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="124569c5-b186-461a-8d4c-6f32c13559aa" containerName="main" Oct 30 06:31:22.272720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.272634 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4230847-8bb3-48c1-99b6-550fa76c4b10" containerName="cleanup" Oct 30 06:31:22.272720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.272641 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4230847-8bb3-48c1-99b6-550fa76c4b10" containerName="cleanup" Oct 30 06:31:22.272720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.272647 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4230847-8bb3-48c1-99b6-550fa76c4b10" containerName="cleanup" Oct 30 06:31:22.276388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.276371 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-c172b3" Oct 30 06:31:22.292064 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.292041 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p6zmk\"" Oct 30 06:31:22.294298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.294274 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-c172b3"] Oct 30 06:31:22.335398 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.335361 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7ljb\" (UniqueName: \"kubernetes.io/projected/ad0d9989-5acf-4c56-a080-c5d063c9fde8-kube-api-access-n7ljb\") pod \"test-curl-7315-c172b3\" (UID: \"ad0d9989-5acf-4c56-a080-c5d063c9fde8\") " pod="opendatahub/test-curl-7315-c172b3" Oct 30 06:31:22.437277 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.437240 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7ljb\" (UniqueName: \"kubernetes.io/projected/ad0d9989-5acf-4c56-a080-c5d063c9fde8-kube-api-access-n7ljb\") pod \"test-curl-7315-c172b3\" (UID: \"ad0d9989-5acf-4c56-a080-c5d063c9fde8\") " pod="opendatahub/test-curl-7315-c172b3" Oct 30 06:31:22.460588 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.460560 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7ljb\" (UniqueName: \"kubernetes.io/projected/ad0d9989-5acf-4c56-a080-c5d063c9fde8-kube-api-access-n7ljb\") pod \"test-curl-7315-c172b3\" (UID: \"ad0d9989-5acf-4c56-a080-c5d063c9fde8\") " pod="opendatahub/test-curl-7315-c172b3" Oct 30 06:31:22.589778 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.589686 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-c172b3" Oct 30 06:31:22.897541 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:31:22.897505 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad0d9989_5acf_4c56_a080_c5d063c9fde8.slice/crio-b8f2aa54a10121c565ef244bfdd7702fe0a48d8e07763138f6cd780882fb06fe WatchSource:0}: Error finding container b8f2aa54a10121c565ef244bfdd7702fe0a48d8e07763138f6cd780882fb06fe: Status 404 returned error can't find the container with id b8f2aa54a10121c565ef244bfdd7702fe0a48d8e07763138f6cd780882fb06fe Oct 30 06:31:22.901008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:22.900985 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-c172b3"] Oct 30 06:31:23.099180 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:23.099141 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-c172b3" event={"ID":"ad0d9989-5acf-4c56-a080-c5d063c9fde8","Type":"ContainerStarted","Data":"d6ec741b6ea8a282cbd646419555819cd1862b58509a05999b5926ee9d726822"} Oct 30 06:31:23.099374 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:23.099189 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-c172b3" event={"ID":"ad0d9989-5acf-4c56-a080-c5d063c9fde8","Type":"ContainerStarted","Data":"b8f2aa54a10121c565ef244bfdd7702fe0a48d8e07763138f6cd780882fb06fe"} Oct 30 06:31:23.138683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:23.138621 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7315-c172b3" podStartSLOduration=1.138605976 podStartE2EDuration="1.138605976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:31:23.136623602 +0000 UTC m=+2249.645643491" watchObservedRunningTime="2026-10-30 06:31:23.138605976 +0000 UTC m=+2249.647625862" Oct 30 06:31:23.259107 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:23.259025 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:24.005189 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.005158 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7315-c172b3"] Oct 30 06:31:24.031237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.011548 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7315-c172b3"] Oct 30 06:31:24.171338 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.171306 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7315-9d5726"] Oct 30 06:31:24.171929 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.171913 6999 cpu_manager.go:414] "containerMap: removing container" podUID="e4230847-8bb3-48c1-99b6-550fa76c4b10" containerName="cleanup" Oct 30 06:31:24.171978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.171931 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4230847-8bb3-48c1-99b6-550fa76c4b10" containerName="cleanup" Oct 30 06:31:24.171978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.171961 6999 cpu_manager.go:414] "containerMap: removing container" podUID="ad0d9989-5acf-4c56-a080-c5d063c9fde8" containerName="test-curl-7315-c172b3" Oct 30 06:31:24.171978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.171965 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad0d9989-5acf-4c56-a080-c5d063c9fde8" containerName="test-curl-7315-c172b3" Oct 30 06:31:24.172079 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.172061 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad0d9989-5acf-4c56-a080-c5d063c9fde8" containerName="test-curl-7315-c172b3" Oct 30 06:31:24.175399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.175385 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-9d5726" Oct 30 06:31:24.192516 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.192488 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-9d5726"] Oct 30 06:31:24.218572 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.218530 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:24.263499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.263425 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hzfn\" (UniqueName: \"kubernetes.io/projected/1f548608-b5c6-4bb4-9d54-75f90c8949ee-kube-api-access-8hzfn\") pod \"test-curl-7315-9d5726\" (UID: \"1f548608-b5c6-4bb4-9d54-75f90c8949ee\") " pod="opendatahub/test-curl-7315-9d5726" Oct 30 06:31:24.364604 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.364566 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hzfn\" (UniqueName: \"kubernetes.io/projected/1f548608-b5c6-4bb4-9d54-75f90c8949ee-kube-api-access-8hzfn\") pod \"test-curl-7315-9d5726\" (UID: \"1f548608-b5c6-4bb4-9d54-75f90c8949ee\") " pod="opendatahub/test-curl-7315-9d5726" Oct 30 06:31:24.387774 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.387747 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hzfn\" (UniqueName: \"kubernetes.io/projected/1f548608-b5c6-4bb4-9d54-75f90c8949ee-kube-api-access-8hzfn\") pod \"test-curl-7315-9d5726\" (UID: \"1f548608-b5c6-4bb4-9d54-75f90c8949ee\") " pod="opendatahub/test-curl-7315-9d5726" Oct 30 06:31:24.488232 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.488190 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-9d5726" Oct 30 06:31:24.800412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:24.800378 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-9d5726"] Oct 30 06:31:24.802364 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:31:24.802335 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f548608_b5c6_4bb4_9d54_75f90c8949ee.slice/crio-3b4d2afe940b13a17861bfa94c3aba04d9734909e5dd6b39ac36f3815fe0eda7 WatchSource:0}: Error finding container 3b4d2afe940b13a17861bfa94c3aba04d9734909e5dd6b39ac36f3815fe0eda7: Status 404 returned error can't find the container with id 3b4d2afe940b13a17861bfa94c3aba04d9734909e5dd6b39ac36f3815fe0eda7 Oct 30 06:31:25.056949 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:25.056863 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:25.113986 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:25.113935 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-9d5726" event={"ID":"1f548608-b5c6-4bb4-9d54-75f90c8949ee","Type":"ContainerStarted","Data":"bc22044155687fd253e4923067e9fc94c96dc6536294e6d89ad76c8c9638fc4a"} Oct 30 06:31:25.113986 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:25.113991 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-9d5726" event={"ID":"1f548608-b5c6-4bb4-9d54-75f90c8949ee","Type":"ContainerStarted","Data":"3b4d2afe940b13a17861bfa94c3aba04d9734909e5dd6b39ac36f3815fe0eda7"} Oct 30 06:31:25.114245 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:25.114126 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7315-c172b3" podUID="ad0d9989-5acf-4c56-a080-c5d063c9fde8" containerName="test-curl-7315-c172b3" containerID="cri-o://d6ec741b6ea8a282cbd646419555819cd1862b58509a05999b5926ee9d726822" gracePeriod=2 Oct 30 06:31:25.163482 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:25.163427 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7315-9d5726" podStartSLOduration=1.163411917 podStartE2EDuration="1.163411917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:31:25.162357408 +0000 UTC m=+2251.671377296" watchObservedRunningTime="2026-10-30 06:31:25.163411917 +0000 UTC m=+2251.672431804" Oct 30 06:31:25.175838 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:25.175806 6999 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7315-c172b3\" is forbidden: User \"system:node:ip-10-0-131-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-226.ec2.internal' and this object" podUID="ad0d9989-5acf-4c56-a080-c5d063c9fde8" pod="opendatahub/test-curl-7315-c172b3" Oct 30 06:31:26.058498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.058468 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7315-9d5726"] Oct 30 06:31:26.071123 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.070864 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7315-9d5726"] Oct 30 06:31:26.223876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.223846 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7315-5c3a6d"] Oct 30 06:31:26.224538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.224522 6999 cpu_manager.go:414] "containerMap: removing container" podUID="1f548608-b5c6-4bb4-9d54-75f90c8949ee" containerName="test-curl-7315-9d5726" Oct 30 06:31:26.224631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.224539 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f548608-b5c6-4bb4-9d54-75f90c8949ee" containerName="test-curl-7315-9d5726" Oct 30 06:31:26.224700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.224690 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f548608-b5c6-4bb4-9d54-75f90c8949ee" containerName="test-curl-7315-9d5726" Oct 30 06:31:26.228555 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.228533 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-5c3a6d" Oct 30 06:31:26.243382 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.243355 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-5c3a6d"] Oct 30 06:31:26.289858 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.289822 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxkkp\" (UniqueName: \"kubernetes.io/projected/d42a46a9-c513-4e11-b4ab-be8d6ca0fd96-kube-api-access-kxkkp\") pod \"test-curl-7315-5c3a6d\" (UID: \"d42a46a9-c513-4e11-b4ab-be8d6ca0fd96\") " pod="opendatahub/test-curl-7315-5c3a6d" Oct 30 06:31:26.391312 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.391284 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxkkp\" (UniqueName: \"kubernetes.io/projected/d42a46a9-c513-4e11-b4ab-be8d6ca0fd96-kube-api-access-kxkkp\") pod \"test-curl-7315-5c3a6d\" (UID: \"d42a46a9-c513-4e11-b4ab-be8d6ca0fd96\") " pod="opendatahub/test-curl-7315-5c3a6d" Oct 30 06:31:26.414967 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.414938 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxkkp\" (UniqueName: \"kubernetes.io/projected/d42a46a9-c513-4e11-b4ab-be8d6ca0fd96-kube-api-access-kxkkp\") pod \"test-curl-7315-5c3a6d\" (UID: \"d42a46a9-c513-4e11-b4ab-be8d6ca0fd96\") " pod="opendatahub/test-curl-7315-5c3a6d" Oct 30 06:31:26.541758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.541728 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-5c3a6d" Oct 30 06:31:26.618444 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.618395 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" podUID="87001573-6955-4231-9fb9-93e335b7d16b" containerName="main" containerID="cri-o://8a778b2dd24f90e8ac0fb128cadd536a3289a4c603f5cf235c62c449038853d6" gracePeriod=45 Oct 30 06:31:26.834508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.834473 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:26.844738 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.844610 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-5c3a6d"] Oct 30 06:31:26.882094 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:26.882070 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:31:27.001059 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.001029 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-dshm\") pod \"87001573-6955-4231-9fb9-93e335b7d16b\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " Oct 30 06:31:27.001256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.001109 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-model-cache\") pod \"87001573-6955-4231-9fb9-93e335b7d16b\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " Oct 30 06:31:27.001256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.001193 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-tmp-dir\") pod \"87001573-6955-4231-9fb9-93e335b7d16b\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " Oct 30 06:31:27.001432 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.001415 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87001573-6955-4231-9fb9-93e335b7d16b-kube-api-access-q7vz9\" (UniqueName: \"kubernetes.io/projected/87001573-6955-4231-9fb9-93e335b7d16b-kube-api-access-q7vz9\") pod \"87001573-6955-4231-9fb9-93e335b7d16b\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " Oct 30 06:31:27.001490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.001411 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-model-cache" pod "87001573-6955-4231-9fb9-93e335b7d16b" (UID: "87001573-6955-4231-9fb9-93e335b7d16b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:27.001550 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.001428 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-tmp-dir" pod "87001573-6955-4231-9fb9-93e335b7d16b" (UID: "87001573-6955-4231-9fb9-93e335b7d16b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:27.001550 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.001475 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87001573-6955-4231-9fb9-93e335b7d16b-tls-certs\" (UniqueName: \"kubernetes.io/secret/87001573-6955-4231-9fb9-93e335b7d16b-tls-certs\") pod \"87001573-6955-4231-9fb9-93e335b7d16b\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " Oct 30 06:31:27.001649 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.001593 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-home\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-home\") pod \"87001573-6955-4231-9fb9-93e335b7d16b\" (UID: \"87001573-6955-4231-9fb9-93e335b7d16b\") " Oct 30 06:31:27.001978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.001874 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-home" pod "87001573-6955-4231-9fb9-93e335b7d16b" (UID: "87001573-6955-4231-9fb9-93e335b7d16b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:27.002263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.002245 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:27.002263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.002264 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:27.002421 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.002273 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:27.003595 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.003569 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-dshm" pod "87001573-6955-4231-9fb9-93e335b7d16b" (UID: "87001573-6955-4231-9fb9-93e335b7d16b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:27.003680 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.003614 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87001573-6955-4231-9fb9-93e335b7d16b-tls-certs" pod "87001573-6955-4231-9fb9-93e335b7d16b" (UID: "87001573-6955-4231-9fb9-93e335b7d16b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:31:27.003680 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.003619 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87001573-6955-4231-9fb9-93e335b7d16b-kube-api-access-q7vz9" pod "87001573-6955-4231-9fb9-93e335b7d16b" (UID: "87001573-6955-4231-9fb9-93e335b7d16b"). InnerVolumeSpecName "kube-api-access-q7vz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:27.103930 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.103902 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7vz9\" (UniqueName: \"kubernetes.io/projected/87001573-6955-4231-9fb9-93e335b7d16b-kube-api-access-q7vz9\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:27.103930 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.103925 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87001573-6955-4231-9fb9-93e335b7d16b-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:27.103930 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.103936 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87001573-6955-4231-9fb9-93e335b7d16b-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:27.130833 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.130804 6999 generic.go:356] "Generic (PLEG): container finished" podID="87001573-6955-4231-9fb9-93e335b7d16b" containerID="8a778b2dd24f90e8ac0fb128cadd536a3289a4c603f5cf235c62c449038853d6" exitCode=0 Oct 30 06:31:27.130927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.130851 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" Oct 30 06:31:27.130927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.130891 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" event={"ID":"87001573-6955-4231-9fb9-93e335b7d16b","Type":"ContainerDied","Data":"8a778b2dd24f90e8ac0fb128cadd536a3289a4c603f5cf235c62c449038853d6"} Oct 30 06:31:27.131015 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.130937 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz" event={"ID":"87001573-6955-4231-9fb9-93e335b7d16b","Type":"ContainerDied","Data":"bcca1145f156eff915fc688fb3af6afa08e4ce372e09dd14820ae5d831b1ef00"} Oct 30 06:31:27.131015 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.130956 6999 scope.go:122] "RemoveContainer" containerID="8a778b2dd24f90e8ac0fb128cadd536a3289a4c603f5cf235c62c449038853d6" Oct 30 06:31:27.133083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.133054 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-5c3a6d" event={"ID":"d42a46a9-c513-4e11-b4ab-be8d6ca0fd96","Type":"ContainerStarted","Data":"1de1784e6b205ae15a353858233d4b1a3dbbd9d38ef1d105133052c0c738f62a"} Oct 30 06:31:27.133202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.133092 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-5c3a6d" event={"ID":"d42a46a9-c513-4e11-b4ab-be8d6ca0fd96","Type":"ContainerStarted","Data":"089bfa87e372eb9bfe84720148d44b277e3831f9c6660d219f0c633840bef117"} Oct 30 06:31:27.133202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.133191 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7315-9d5726" podUID="1f548608-b5c6-4bb4-9d54-75f90c8949ee" containerName="test-curl-7315-9d5726" containerID="cri-o://bc22044155687fd253e4923067e9fc94c96dc6536294e6d89ad76c8c9638fc4a" gracePeriod=2 Oct 30 06:31:27.159410 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:27.159340 6999 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7315-9d5726\" is forbidden: User \"system:node:ip-10-0-131-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-226.ec2.internal' and this object" podUID="1f548608-b5c6-4bb4-9d54-75f90c8949ee" pod="opendatahub/test-curl-7315-9d5726" Oct 30 06:31:27.167001 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.166979 6999 scope.go:122] "RemoveContainer" containerID="8a778b2dd24f90e8ac0fb128cadd536a3289a4c603f5cf235c62c449038853d6" Oct 30 06:31:27.167357 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:27.167327 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a778b2dd24f90e8ac0fb128cadd536a3289a4c603f5cf235c62c449038853d6\": container with ID starting with 8a778b2dd24f90e8ac0fb128cadd536a3289a4c603f5cf235c62c449038853d6 not found: ID does not exist" containerID="8a778b2dd24f90e8ac0fb128cadd536a3289a4c603f5cf235c62c449038853d6" Oct 30 06:31:27.167442 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.167369 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a778b2dd24f90e8ac0fb128cadd536a3289a4c603f5cf235c62c449038853d6"} err="failed to get container status \"8a778b2dd24f90e8ac0fb128cadd536a3289a4c603f5cf235c62c449038853d6\": rpc error: code = NotFound desc = could not find container \"8a778b2dd24f90e8ac0fb128cadd536a3289a4c603f5cf235c62c449038853d6\": container with ID starting with 8a778b2dd24f90e8ac0fb128cadd536a3289a4c603f5cf235c62c449038853d6 not found: ID does not exist" Oct 30 06:31:27.198776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.198730 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7315-5c3a6d" podStartSLOduration=1.198716745 podStartE2EDuration="1.198716745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:31:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:31:27.196856689 +0000 UTC m=+2253.705876578" watchObservedRunningTime="2026-10-30 06:31:27.198716745 +0000 UTC m=+2253.707736632" Oct 30 06:31:27.233528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.233505 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz"] Oct 30 06:31:27.236899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.236880 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-adcdcb/auth-test-model-adcdcb-kserve-58c8f94cd9-4g7qz"] Oct 30 06:31:27.244951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.244933 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-c172b3" Oct 30 06:31:27.247421 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:27.247400 6999 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7315-9d5726\" is forbidden: User \"system:node:ip-10-0-131-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-226.ec2.internal' and this object" podUID="1f548608-b5c6-4bb4-9d54-75f90c8949ee" pod="opendatahub/test-curl-7315-9d5726" Oct 30 06:31:27.260733 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:27.260705 6999 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7315-9d5726\" is forbidden: User \"system:node:ip-10-0-131-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-226.ec2.internal' and this object" podUID="1f548608-b5c6-4bb4-9d54-75f90c8949ee" pod="opendatahub/test-curl-7315-9d5726" Oct 30 06:31:27.305909 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.305889 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad0d9989-5acf-4c56-a080-c5d063c9fde8-kube-api-access-n7ljb\" (UniqueName: \"kubernetes.io/projected/ad0d9989-5acf-4c56-a080-c5d063c9fde8-kube-api-access-n7ljb\") pod \"ad0d9989-5acf-4c56-a080-c5d063c9fde8\" (UID: \"ad0d9989-5acf-4c56-a080-c5d063c9fde8\") " Oct 30 06:31:27.308030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.308008 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0d9989-5acf-4c56-a080-c5d063c9fde8-kube-api-access-n7ljb" pod "ad0d9989-5acf-4c56-a080-c5d063c9fde8" (UID: "ad0d9989-5acf-4c56-a080-c5d063c9fde8"). InnerVolumeSpecName "kube-api-access-n7ljb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:27.407783 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.407760 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7ljb\" (UniqueName: \"kubernetes.io/projected/ad0d9989-5acf-4c56-a080-c5d063c9fde8-kube-api-access-n7ljb\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:27.609895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.609866 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87001573-6955-4231-9fb9-93e335b7d16b" path="/var/lib/kubelet/pods/87001573-6955-4231-9fb9-93e335b7d16b/volumes" Oct 30 06:31:27.610303 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:27.610285 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad0d9989-5acf-4c56-a080-c5d063c9fde8" path="/var/lib/kubelet/pods/ad0d9989-5acf-4c56-a080-c5d063c9fde8/volumes" Oct 30 06:31:28.035174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.035136 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7315-5c3a6d"] Oct 30 06:31:28.038975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.038950 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7315-5c3a6d"] Oct 30 06:31:28.143728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.143686 6999 generic.go:356] "Generic (PLEG): container finished" podID="ad0d9989-5acf-4c56-a080-c5d063c9fde8" containerID="d6ec741b6ea8a282cbd646419555819cd1862b58509a05999b5926ee9d726822" exitCode=137 Oct 30 06:31:28.144186 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.143762 6999 scope.go:122] "RemoveContainer" containerID="d6ec741b6ea8a282cbd646419555819cd1862b58509a05999b5926ee9d726822" Oct 30 06:31:28.144186 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.143771 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-c172b3" Oct 30 06:31:28.153717 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.153695 6999 scope.go:122] "RemoveContainer" containerID="d6ec741b6ea8a282cbd646419555819cd1862b58509a05999b5926ee9d726822" Oct 30 06:31:28.154024 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:28.153996 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6ec741b6ea8a282cbd646419555819cd1862b58509a05999b5926ee9d726822\": container with ID starting with d6ec741b6ea8a282cbd646419555819cd1862b58509a05999b5926ee9d726822 not found: ID does not exist" containerID="d6ec741b6ea8a282cbd646419555819cd1862b58509a05999b5926ee9d726822" Oct 30 06:31:28.154103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.154035 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6ec741b6ea8a282cbd646419555819cd1862b58509a05999b5926ee9d726822"} err="failed to get container status \"d6ec741b6ea8a282cbd646419555819cd1862b58509a05999b5926ee9d726822\": rpc error: code = NotFound desc = could not find container \"d6ec741b6ea8a282cbd646419555819cd1862b58509a05999b5926ee9d726822\": container with ID starting with d6ec741b6ea8a282cbd646419555819cd1862b58509a05999b5926ee9d726822 not found: ID does not exist" Oct 30 06:31:28.204915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.204879 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7315-69263c"] Oct 30 06:31:28.205497 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.205483 6999 cpu_manager.go:414] "containerMap: removing container" podUID="87001573-6955-4231-9fb9-93e335b7d16b" containerName="main" Oct 30 06:31:28.205550 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.205498 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87001573-6955-4231-9fb9-93e335b7d16b" containerName="main" Oct 30 06:31:28.205550 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.205526 6999 cpu_manager.go:414] "containerMap: removing container" podUID="d42a46a9-c513-4e11-b4ab-be8d6ca0fd96" containerName="test-curl-7315-5c3a6d" Oct 30 06:31:28.205550 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.205531 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d42a46a9-c513-4e11-b4ab-be8d6ca0fd96" containerName="test-curl-7315-5c3a6d" Oct 30 06:31:28.205654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.205626 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="87001573-6955-4231-9fb9-93e335b7d16b" containerName="main" Oct 30 06:31:28.205654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.205639 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="d42a46a9-c513-4e11-b4ab-be8d6ca0fd96" containerName="test-curl-7315-5c3a6d" Oct 30 06:31:28.210298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.210282 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-69263c" Oct 30 06:31:28.227350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.227330 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-69263c"] Oct 30 06:31:28.319132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.319053 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxlz2\" (UniqueName: \"kubernetes.io/projected/7e753ce8-3050-4563-92bc-beedc93f95c0-kube-api-access-sxlz2\") pod \"test-curl-7315-69263c\" (UID: \"7e753ce8-3050-4563-92bc-beedc93f95c0\") " pod="opendatahub/test-curl-7315-69263c" Oct 30 06:31:28.420687 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.420659 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxlz2\" (UniqueName: \"kubernetes.io/projected/7e753ce8-3050-4563-92bc-beedc93f95c0-kube-api-access-sxlz2\") pod \"test-curl-7315-69263c\" (UID: \"7e753ce8-3050-4563-92bc-beedc93f95c0\") " pod="opendatahub/test-curl-7315-69263c" Oct 30 06:31:28.443907 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.443873 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxlz2\" (UniqueName: \"kubernetes.io/projected/7e753ce8-3050-4563-92bc-beedc93f95c0-kube-api-access-sxlz2\") pod \"test-curl-7315-69263c\" (UID: \"7e753ce8-3050-4563-92bc-beedc93f95c0\") " pod="opendatahub/test-curl-7315-69263c" Oct 30 06:31:28.522003 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.521976 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-69263c" Oct 30 06:31:28.622789 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.622758 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:28.826782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:28.826716 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7315-69263c"] Oct 30 06:31:28.828363 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:31:28.828333 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e753ce8_3050_4563_92bc_beedc93f95c0.slice/crio-950ce701dc037f0b8ba3f677fd25a2fdab6d984f7437200114320e36462a9893 WatchSource:0}: Error finding container 950ce701dc037f0b8ba3f677fd25a2fdab6d984f7437200114320e36462a9893: Status 404 returned error can't find the container with id 950ce701dc037f0b8ba3f677fd25a2fdab6d984f7437200114320e36462a9893 Oct 30 06:31:29.154121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:29.154089 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-69263c" event={"ID":"7e753ce8-3050-4563-92bc-beedc93f95c0","Type":"ContainerStarted","Data":"3c4a4eec059fe94a73afe87fe3cb628ff5c54a45ac7a5fb60a65eae464cc7191"} Oct 30 06:31:29.154499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:29.154131 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7315-69263c" event={"ID":"7e753ce8-3050-4563-92bc-beedc93f95c0","Type":"ContainerStarted","Data":"950ce701dc037f0b8ba3f677fd25a2fdab6d984f7437200114320e36462a9893"} Oct 30 06:31:29.154499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:29.154291 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7315-5c3a6d" podUID="d42a46a9-c513-4e11-b4ab-be8d6ca0fd96" containerName="test-curl-7315-5c3a6d" containerID="cri-o://1de1784e6b205ae15a353858233d4b1a3dbbd9d38ef1d105133052c0c738f62a" gracePeriod=2 Oct 30 06:31:29.192108 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:29.192054 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7315-69263c" podStartSLOduration=1.192040296 podStartE2EDuration="1.192040296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:31:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:31:29.191920408 +0000 UTC m=+2255.700940296" watchObservedRunningTime="2026-10-30 06:31:29.192040296 +0000 UTC m=+2255.701060182" Oct 30 06:31:29.205294 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:29.205262 6999 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7315-5c3a6d\" is forbidden: User \"system:node:ip-10-0-131-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-226.ec2.internal' and this object" podUID="d42a46a9-c513-4e11-b4ab-be8d6ca0fd96" pod="opendatahub/test-curl-7315-5c3a6d" Oct 30 06:31:29.319663 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:29.319636 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-9d5726" Oct 30 06:31:29.336721 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:29.336688 6999 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7315-5c3a6d\" is forbidden: User \"system:node:ip-10-0-131-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-226.ec2.internal' and this object" podUID="d42a46a9-c513-4e11-b4ab-be8d6ca0fd96" pod="opendatahub/test-curl-7315-5c3a6d" Oct 30 06:31:29.433941 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:29.433865 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f548608-b5c6-4bb4-9d54-75f90c8949ee-kube-api-access-8hzfn\" (UniqueName: \"kubernetes.io/projected/1f548608-b5c6-4bb4-9d54-75f90c8949ee-kube-api-access-8hzfn\") pod \"1f548608-b5c6-4bb4-9d54-75f90c8949ee\" (UID: \"1f548608-b5c6-4bb4-9d54-75f90c8949ee\") " Oct 30 06:31:29.436243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:29.436203 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f548608-b5c6-4bb4-9d54-75f90c8949ee-kube-api-access-8hzfn" pod "1f548608-b5c6-4bb4-9d54-75f90c8949ee" (UID: "1f548608-b5c6-4bb4-9d54-75f90c8949ee"). InnerVolumeSpecName "kube-api-access-8hzfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:29.535551 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:29.535519 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hzfn\" (UniqueName: \"kubernetes.io/projected/1f548608-b5c6-4bb4-9d54-75f90c8949ee-kube-api-access-8hzfn\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:29.609879 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:29.609839 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f548608-b5c6-4bb4-9d54-75f90c8949ee" path="/var/lib/kubelet/pods/1f548608-b5c6-4bb4-9d54-75f90c8949ee/volumes" Oct 30 06:31:30.076381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:30.076342 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7315-69263c"] Oct 30 06:31:30.080880 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:30.080853 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7315-69263c"] Oct 30 06:31:30.163994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:30.163955 6999 generic.go:356] "Generic (PLEG): container finished" podID="1f548608-b5c6-4bb4-9d54-75f90c8949ee" containerID="bc22044155687fd253e4923067e9fc94c96dc6536294e6d89ad76c8c9638fc4a" exitCode=137 Oct 30 06:31:30.164510 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:30.164026 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-9d5726" Oct 30 06:31:30.164510 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:30.164025 6999 scope.go:122] "RemoveContainer" containerID="bc22044155687fd253e4923067e9fc94c96dc6536294e6d89ad76c8c9638fc4a" Oct 30 06:31:30.175629 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:30.175607 6999 scope.go:122] "RemoveContainer" containerID="bc22044155687fd253e4923067e9fc94c96dc6536294e6d89ad76c8c9638fc4a" Oct 30 06:31:30.176017 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:30.175986 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc22044155687fd253e4923067e9fc94c96dc6536294e6d89ad76c8c9638fc4a\": container with ID starting with bc22044155687fd253e4923067e9fc94c96dc6536294e6d89ad76c8c9638fc4a not found: ID does not exist" containerID="bc22044155687fd253e4923067e9fc94c96dc6536294e6d89ad76c8c9638fc4a" Oct 30 06:31:30.176114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:30.176027 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc22044155687fd253e4923067e9fc94c96dc6536294e6d89ad76c8c9638fc4a"} err="failed to get container status \"bc22044155687fd253e4923067e9fc94c96dc6536294e6d89ad76c8c9638fc4a\": rpc error: code = NotFound desc = could not find container \"bc22044155687fd253e4923067e9fc94c96dc6536294e6d89ad76c8c9638fc4a\": container with ID starting with bc22044155687fd253e4923067e9fc94c96dc6536294e6d89ad76c8c9638fc4a not found: ID does not exist" Oct 30 06:31:30.465402 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:30.465325 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:31.171553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:31.171527 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7315-69263c" podUID="7e753ce8-3050-4563-92bc-beedc93f95c0" containerName="test-curl-7315-69263c" containerID="cri-o://3c4a4eec059fe94a73afe87fe3cb628ff5c54a45ac7a5fb60a65eae464cc7191" gracePeriod=2 Oct 30 06:31:31.185068 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:31.185039 6999 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7315-69263c\" is forbidden: User \"system:node:ip-10-0-131-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-226.ec2.internal' and this object" podUID="7e753ce8-3050-4563-92bc-beedc93f95c0" pod="opendatahub/test-curl-7315-69263c" Oct 30 06:31:31.372358 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:31.372334 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-5c3a6d" Oct 30 06:31:31.385847 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:31.385815 6999 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7315-69263c\" is forbidden: User \"system:node:ip-10-0-131-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-226.ec2.internal' and this object" podUID="7e753ce8-3050-4563-92bc-beedc93f95c0" pod="opendatahub/test-curl-7315-69263c" Oct 30 06:31:31.390977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:31.390940 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:31.462900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:31.462827 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42a46a9-c513-4e11-b4ab-be8d6ca0fd96-kube-api-access-kxkkp\" (UniqueName: \"kubernetes.io/projected/d42a46a9-c513-4e11-b4ab-be8d6ca0fd96-kube-api-access-kxkkp\") pod \"d42a46a9-c513-4e11-b4ab-be8d6ca0fd96\" (UID: \"d42a46a9-c513-4e11-b4ab-be8d6ca0fd96\") " Oct 30 06:31:31.465191 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:31.465165 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42a46a9-c513-4e11-b4ab-be8d6ca0fd96-kube-api-access-kxkkp" pod "d42a46a9-c513-4e11-b4ab-be8d6ca0fd96" (UID: "d42a46a9-c513-4e11-b4ab-be8d6ca0fd96"). InnerVolumeSpecName "kube-api-access-kxkkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:31.564767 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:31.564730 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxkkp\" (UniqueName: \"kubernetes.io/projected/d42a46a9-c513-4e11-b4ab-be8d6ca0fd96-kube-api-access-kxkkp\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:31.610858 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:31.610830 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d42a46a9-c513-4e11-b4ab-be8d6ca0fd96" path="/var/lib/kubelet/pods/d42a46a9-c513-4e11-b4ab-be8d6ca0fd96/volumes" Oct 30 06:31:31.873285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:31.873252 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:32.066768 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:32.066737 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:32.179498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:32.179413 6999 generic.go:356] "Generic (PLEG): container finished" podID="d42a46a9-c513-4e11-b4ab-be8d6ca0fd96" containerID="1de1784e6b205ae15a353858233d4b1a3dbbd9d38ef1d105133052c0c738f62a" exitCode=137 Oct 30 06:31:32.179498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:32.179474 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-5c3a6d" Oct 30 06:31:32.180021 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:32.179480 6999 scope.go:122] "RemoveContainer" containerID="1de1784e6b205ae15a353858233d4b1a3dbbd9d38ef1d105133052c0c738f62a" Oct 30 06:31:32.190462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:32.190438 6999 scope.go:122] "RemoveContainer" containerID="1de1784e6b205ae15a353858233d4b1a3dbbd9d38ef1d105133052c0c738f62a" Oct 30 06:31:32.190770 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:32.190746 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de1784e6b205ae15a353858233d4b1a3dbbd9d38ef1d105133052c0c738f62a\": container with ID starting with 1de1784e6b205ae15a353858233d4b1a3dbbd9d38ef1d105133052c0c738f62a not found: ID does not exist" containerID="1de1784e6b205ae15a353858233d4b1a3dbbd9d38ef1d105133052c0c738f62a" Oct 30 06:31:32.190837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:32.190781 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de1784e6b205ae15a353858233d4b1a3dbbd9d38ef1d105133052c0c738f62a"} err="failed to get container status \"1de1784e6b205ae15a353858233d4b1a3dbbd9d38ef1d105133052c0c738f62a\": rpc error: code = NotFound desc = could not find container \"1de1784e6b205ae15a353858233d4b1a3dbbd9d38ef1d105133052c0c738f62a\": container with ID starting with 1de1784e6b205ae15a353858233d4b1a3dbbd9d38ef1d105133052c0c738f62a not found: ID does not exist" Oct 30 06:31:32.317104 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:32.317070 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:32.490300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:32.485733 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:32.651900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:32.651849 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:33.400995 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:33.400943 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-69263c" Oct 30 06:31:33.517266 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:33.517176 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e753ce8-3050-4563-92bc-beedc93f95c0-kube-api-access-sxlz2\" (UniqueName: \"kubernetes.io/projected/7e753ce8-3050-4563-92bc-beedc93f95c0-kube-api-access-sxlz2\") pod \"7e753ce8-3050-4563-92bc-beedc93f95c0\" (UID: \"7e753ce8-3050-4563-92bc-beedc93f95c0\") " Oct 30 06:31:33.519397 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:33.519365 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e753ce8-3050-4563-92bc-beedc93f95c0-kube-api-access-sxlz2" pod "7e753ce8-3050-4563-92bc-beedc93f95c0" (UID: "7e753ce8-3050-4563-92bc-beedc93f95c0"). InnerVolumeSpecName "kube-api-access-sxlz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:33.611066 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:33.611032 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e753ce8-3050-4563-92bc-beedc93f95c0" path="/var/lib/kubelet/pods/7e753ce8-3050-4563-92bc-beedc93f95c0/volumes" Oct 30 06:31:33.618910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:33.618882 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxlz2\" (UniqueName: \"kubernetes.io/projected/7e753ce8-3050-4563-92bc-beedc93f95c0-kube-api-access-sxlz2\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:34.063697 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:34.063643 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:34.148544 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:34.148507 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:31:34.196118 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:34.196084 6999 generic.go:356] "Generic (PLEG): container finished" podID="7e753ce8-3050-4563-92bc-beedc93f95c0" containerID="3c4a4eec059fe94a73afe87fe3cb628ff5c54a45ac7a5fb60a65eae464cc7191" exitCode=137 Oct 30 06:31:34.196421 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:34.196145 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7315-69263c" Oct 30 06:31:34.196421 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:34.196176 6999 scope.go:122] "RemoveContainer" containerID="3c4a4eec059fe94a73afe87fe3cb628ff5c54a45ac7a5fb60a65eae464cc7191" Oct 30 06:31:34.206059 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:34.206041 6999 scope.go:122] "RemoveContainer" containerID="3c4a4eec059fe94a73afe87fe3cb628ff5c54a45ac7a5fb60a65eae464cc7191" Oct 30 06:31:34.206421 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:34.206398 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c4a4eec059fe94a73afe87fe3cb628ff5c54a45ac7a5fb60a65eae464cc7191\": container with ID starting with 3c4a4eec059fe94a73afe87fe3cb628ff5c54a45ac7a5fb60a65eae464cc7191 not found: ID does not exist" containerID="3c4a4eec059fe94a73afe87fe3cb628ff5c54a45ac7a5fb60a65eae464cc7191" Oct 30 06:31:34.206535 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:34.206429 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c4a4eec059fe94a73afe87fe3cb628ff5c54a45ac7a5fb60a65eae464cc7191"} err="failed to get container status \"3c4a4eec059fe94a73afe87fe3cb628ff5c54a45ac7a5fb60a65eae464cc7191\": rpc error: code = NotFound desc = could not find container \"3c4a4eec059fe94a73afe87fe3cb628ff5c54a45ac7a5fb60a65eae464cc7191\": container with ID starting with 3c4a4eec059fe94a73afe87fe3cb628ff5c54a45ac7a5fb60a65eae464cc7191 not found: ID does not exist" Oct 30 06:31:36.273566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:36.273533 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:36.435996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:36.435966 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:36.583427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:36.583329 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:36.743177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:36.743139 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:36.884014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:36.883984 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:36.952431 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:36.952389 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" podUID="dd1af9cf-0a72-4d17-9544-c1e86f081185" containerName="main" containerID="cri-o://51b3d8394b7264d202c202b862963391bb107efeaa3a434e7d5266c417480f23" gracePeriod=45 Oct 30 06:31:37.210083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.210060 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:31:37.217806 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.217778 6999 generic.go:356] "Generic (PLEG): container finished" podID="dd1af9cf-0a72-4d17-9544-c1e86f081185" containerID="51b3d8394b7264d202c202b862963391bb107efeaa3a434e7d5266c417480f23" exitCode=0 Oct 30 06:31:37.217940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.217824 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" Oct 30 06:31:37.217940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.217865 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" event={"ID":"dd1af9cf-0a72-4d17-9544-c1e86f081185","Type":"ContainerDied","Data":"51b3d8394b7264d202c202b862963391bb107efeaa3a434e7d5266c417480f23"} Oct 30 06:31:37.217940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.217905 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95" event={"ID":"dd1af9cf-0a72-4d17-9544-c1e86f081185","Type":"ContainerDied","Data":"aa17401aa71362f8819c5c30d10dd538f0b6f983a41124cf209719e42ffab39e"} Oct 30 06:31:37.217940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.217928 6999 scope.go:122] "RemoveContainer" containerID="51b3d8394b7264d202c202b862963391bb107efeaa3a434e7d5266c417480f23" Oct 30 06:31:37.229607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.229584 6999 scope.go:122] "RemoveContainer" containerID="51b3d8394b7264d202c202b862963391bb107efeaa3a434e7d5266c417480f23" Oct 30 06:31:37.229920 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:31:37.229894 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51b3d8394b7264d202c202b862963391bb107efeaa3a434e7d5266c417480f23\": container with ID starting with 51b3d8394b7264d202c202b862963391bb107efeaa3a434e7d5266c417480f23 not found: ID does not exist" containerID="51b3d8394b7264d202c202b862963391bb107efeaa3a434e7d5266c417480f23" Oct 30 06:31:37.230009 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.229926 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51b3d8394b7264d202c202b862963391bb107efeaa3a434e7d5266c417480f23"} err="failed to get container status \"51b3d8394b7264d202c202b862963391bb107efeaa3a434e7d5266c417480f23\": rpc error: code = NotFound desc = could not find container \"51b3d8394b7264d202c202b862963391bb107efeaa3a434e7d5266c417480f23\": container with ID starting with 51b3d8394b7264d202c202b862963391bb107efeaa3a434e7d5266c417480f23 not found: ID does not exist" Oct 30 06:31:37.273702 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.273674 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd1af9cf-0a72-4d17-9544-c1e86f081185-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1af9cf-0a72-4d17-9544-c1e86f081185-tls-certs\") pod \"dd1af9cf-0a72-4d17-9544-c1e86f081185\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " Oct 30 06:31:37.274062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.273834 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-dshm\") pod \"dd1af9cf-0a72-4d17-9544-c1e86f081185\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " Oct 30 06:31:37.274062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.273864 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1af9cf-0a72-4d17-9544-c1e86f081185-kube-api-access-smsqg\" (UniqueName: \"kubernetes.io/projected/dd1af9cf-0a72-4d17-9544-c1e86f081185-kube-api-access-smsqg\") pod \"dd1af9cf-0a72-4d17-9544-c1e86f081185\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " Oct 30 06:31:37.274062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.273893 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-model-cache\") pod \"dd1af9cf-0a72-4d17-9544-c1e86f081185\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " Oct 30 06:31:37.274062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.273927 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-home\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-home\") pod \"dd1af9cf-0a72-4d17-9544-c1e86f081185\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " Oct 30 06:31:37.274062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.273997 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-tmp-dir\") pod \"dd1af9cf-0a72-4d17-9544-c1e86f081185\" (UID: \"dd1af9cf-0a72-4d17-9544-c1e86f081185\") " Oct 30 06:31:37.274388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.274147 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-model-cache" pod "dd1af9cf-0a72-4d17-9544-c1e86f081185" (UID: "dd1af9cf-0a72-4d17-9544-c1e86f081185"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:37.274388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.274249 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-home" pod "dd1af9cf-0a72-4d17-9544-c1e86f081185" (UID: "dd1af9cf-0a72-4d17-9544-c1e86f081185"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:37.274388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.274337 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-tmp-dir" pod "dd1af9cf-0a72-4d17-9544-c1e86f081185" (UID: "dd1af9cf-0a72-4d17-9544-c1e86f081185"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:37.274814 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.274790 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:37.274887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.274820 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:37.274887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.274835 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:37.276201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.276177 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-dshm" pod "dd1af9cf-0a72-4d17-9544-c1e86f081185" (UID: "dd1af9cf-0a72-4d17-9544-c1e86f081185"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:31:37.276201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.276190 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd1af9cf-0a72-4d17-9544-c1e86f081185-tls-certs" pod "dd1af9cf-0a72-4d17-9544-c1e86f081185" (UID: "dd1af9cf-0a72-4d17-9544-c1e86f081185"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:31:37.276366 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.276203 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1af9cf-0a72-4d17-9544-c1e86f081185-kube-api-access-smsqg" pod "dd1af9cf-0a72-4d17-9544-c1e86f081185" (UID: "dd1af9cf-0a72-4d17-9544-c1e86f081185"). InnerVolumeSpecName "kube-api-access-smsqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:37.376245 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.376179 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1af9cf-0a72-4d17-9544-c1e86f081185-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:37.376245 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.376209 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd1af9cf-0a72-4d17-9544-c1e86f081185-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:37.376245 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.376235 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smsqg\" (UniqueName: \"kubernetes.io/projected/dd1af9cf-0a72-4d17-9544-c1e86f081185-kube-api-access-smsqg\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:31:37.566620 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.566591 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95"] Oct 30 06:31:37.572847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.572822 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-fa5a69/auth-test-model-fa5a69-kserve-56d444fdb9-xtp95"] Oct 30 06:31:37.609515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:37.609474 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd1af9cf-0a72-4d17-9544-c1e86f081185" path="/var/lib/kubelet/pods/dd1af9cf-0a72-4d17-9544-c1e86f081185/volumes" Oct 30 06:31:38.128427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:38.128390 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:38.271551 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:38.271515 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:38.423996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:38.423914 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:39.789582 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:39.789555 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:39.945883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:39.945850 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:40.264018 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:40.263983 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:41.987994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:41.987963 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:43.796952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:43.796917 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:45.643976 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:45.643945 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:47.463840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:47.463779 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:49.234207 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:49.234171 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:51.029916 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:51.029883 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:52.875889 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:52.875857 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:54.627537 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:54.627493 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:55.340206 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:55.340161 6999 scope.go:122] "RemoveContainer" containerID="67d0b5fb26b92b30fde2debcfe712d8df1c1357e01a6a2db8c5de6d7185d8148" Oct 30 06:31:56.425955 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:56.425924 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:58.144378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:58.144318 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:58.677156 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:58.677121 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:31:59.933000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:31:59.932968 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:01.556455 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.556426 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb"] Oct 30 06:32:01.557108 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.557086 6999 cpu_manager.go:414] "containerMap: removing container" podUID="dd1af9cf-0a72-4d17-9544-c1e86f081185" containerName="main" Oct 30 06:32:01.557174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.557110 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1af9cf-0a72-4d17-9544-c1e86f081185" containerName="main" Oct 30 06:32:01.557174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.557132 6999 cpu_manager.go:414] "containerMap: removing container" podUID="7e753ce8-3050-4563-92bc-beedc93f95c0" containerName="test-curl-7315-69263c" Oct 30 06:32:01.557174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.557137 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e753ce8-3050-4563-92bc-beedc93f95c0" containerName="test-curl-7315-69263c" Oct 30 06:32:01.557300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.557270 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd1af9cf-0a72-4d17-9544-c1e86f081185" containerName="main" Oct 30 06:32:01.557300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.557285 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e753ce8-3050-4563-92bc-beedc93f95c0" containerName="test-curl-7315-69263c" Oct 30 06:32:01.562085 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.562066 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:01.591839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.591811 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-b8ca49-serving-cert\"" Oct 30 06:32:01.594970 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.594943 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb"] Oct 30 06:32:01.599611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.599585 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p25hn\" (UniqueName: \"kubernetes.io/projected/8111e1b2-a50e-4d15-8efb-00febd24b5e3-kube-api-access-p25hn\") pod \"maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:01.599736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.599649 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8111e1b2-a50e-4d15-8efb-00febd24b5e3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:01.599791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.599754 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8111e1b2-a50e-4d15-8efb-00febd24b5e3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:01.599838 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.599825 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8111e1b2-a50e-4d15-8efb-00febd24b5e3-metrics-certs\") pod \"maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:01.671173 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.671138 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:01.701444 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.701415 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p25hn\" (UniqueName: \"kubernetes.io/projected/8111e1b2-a50e-4d15-8efb-00febd24b5e3-kube-api-access-p25hn\") pod \"maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:01.701606 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.701480 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8111e1b2-a50e-4d15-8efb-00febd24b5e3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:01.701606 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.701514 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8111e1b2-a50e-4d15-8efb-00febd24b5e3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:01.701606 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.701598 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8111e1b2-a50e-4d15-8efb-00febd24b5e3-metrics-certs\") pod \"maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:01.702230 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.702192 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8111e1b2-a50e-4d15-8efb-00febd24b5e3-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:01.704034 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.704013 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8111e1b2-a50e-4d15-8efb-00febd24b5e3-metrics-certs\") pod \"maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:01.704130 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.704036 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8111e1b2-a50e-4d15-8efb-00febd24b5e3-maas-api-tls\") pod \"maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:01.724792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.724762 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p25hn\" (UniqueName: \"kubernetes.io/projected/8111e1b2-a50e-4d15-8efb-00febd24b5e3-kube-api-access-p25hn\") pod \"maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:01.819155 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.819064 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:01.875733 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:01.875706 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:02.060801 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:02.060759 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:02.184650 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:02.184627 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb"] Oct 30 06:32:02.186092 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:32:02.186066 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8111e1b2_a50e_4d15_8efb_00febd24b5e3.slice/crio-a81d1d4dda16a8f67325d03dadb559759c160d9f22717d667d58e011468b2d50 WatchSource:0}: Error finding container a81d1d4dda16a8f67325d03dadb559759c160d9f22717d667d58e011468b2d50: Status 404 returned error can't find the container with id a81d1d4dda16a8f67325d03dadb559759c160d9f22717d667d58e011468b2d50 Oct 30 06:32:02.393441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:02.393404 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" event={"ID":"8111e1b2-a50e-4d15-8efb-00febd24b5e3","Type":"ContainerStarted","Data":"a81d1d4dda16a8f67325d03dadb559759c160d9f22717d667d58e011468b2d50"} Oct 30 06:32:02.419960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:02.419927 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:03.400812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.400775 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" event={"ID":"8111e1b2-a50e-4d15-8efb-00febd24b5e3","Type":"ContainerStarted","Data":"be19481bf527ae593c08c3911eaabc0eb3a0bace69fb39930da0ff18115ba738"} Oct 30 06:32:03.401334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.400940 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:03.444657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.444611 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" podStartSLOduration=2.039098523 podStartE2EDuration="2.444596459s" totalImagesPullingTime="405.497936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:03.443152166 +0000 UTC m=+2289.952172054" watchObservedRunningTime="2026-10-30 06:32:03.444596459 +0000 UTC m=+2289.953616345" Oct 30 06:32:03.458887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.458857 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn"] Oct 30 06:32:03.463064 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.463047 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:03.477368 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.477341 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-2d2923-serving-cert\"" Oct 30 06:32:03.486657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.486623 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn"] Oct 30 06:32:03.528196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.528161 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f523aacd-5b3e-4436-a370-c885f63b23cc-metrics-certs\") pod \"maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:03.528196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.528202 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f523aacd-5b3e-4436-a370-c885f63b23cc-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:03.528486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.528240 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f523aacd-5b3e-4436-a370-c885f63b23cc-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:03.528486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.528415 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndfk4\" (UniqueName: \"kubernetes.io/projected/f523aacd-5b3e-4436-a370-c885f63b23cc-kube-api-access-ndfk4\") pod \"maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:03.581159 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.581130 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:03.629428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.629401 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndfk4\" (UniqueName: \"kubernetes.io/projected/f523aacd-5b3e-4436-a370-c885f63b23cc-kube-api-access-ndfk4\") pod \"maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:03.629627 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.629504 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f523aacd-5b3e-4436-a370-c885f63b23cc-metrics-certs\") pod \"maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:03.629627 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.629546 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f523aacd-5b3e-4436-a370-c885f63b23cc-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:03.629627 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.629571 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f523aacd-5b3e-4436-a370-c885f63b23cc-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:03.630297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.630270 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f523aacd-5b3e-4436-a370-c885f63b23cc-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:03.632269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.632242 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f523aacd-5b3e-4436-a370-c885f63b23cc-metrics-certs\") pod \"maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:03.632362 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.632240 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f523aacd-5b3e-4436-a370-c885f63b23cc-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:03.652765 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.652713 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndfk4\" (UniqueName: \"kubernetes.io/projected/f523aacd-5b3e-4436-a370-c885f63b23cc-kube-api-access-ndfk4\") pod \"maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:03.729883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.729848 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:03.777877 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:03.777546 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:04.089235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:04.089188 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn"] Oct 30 06:32:04.090737 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:32:04.090713 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf523aacd_5b3e_4436_a370_c885f63b23cc.slice/crio-996fe5972bbb3f80cc0344f24d056396d6a0621a963146ac6dd544c1e4cb3ca3 WatchSource:0}: Error finding container 996fe5972bbb3f80cc0344f24d056396d6a0621a963146ac6dd544c1e4cb3ca3: Status 404 returned error can't find the container with id 996fe5972bbb3f80cc0344f24d056396d6a0621a963146ac6dd544c1e4cb3ca3 Oct 30 06:32:04.143158 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:32:04.143124 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:32:04.404829 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:04.404799 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:04.409045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:04.409008 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" event={"ID":"f523aacd-5b3e-4436-a370-c885f63b23cc","Type":"ContainerStarted","Data":"996fe5972bbb3f80cc0344f24d056396d6a0621a963146ac6dd544c1e4cb3ca3"} Oct 30 06:32:05.418440 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:05.418402 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" event={"ID":"f523aacd-5b3e-4436-a370-c885f63b23cc","Type":"ContainerStarted","Data":"f5933cf2d33aabfa70792d57e999686309c7e359e2908f43420093306c486f22"} Oct 30 06:32:05.418885 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:05.418575 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:05.461094 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:05.461041 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" podStartSLOduration=2.102636154 podStartE2EDuration="2.461025513s" totalImagesPullingTime="358.389359ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:05.459826553 +0000 UTC m=+2291.968846440" watchObservedRunningTime="2026-10-30 06:32:05.461025513 +0000 UTC m=+2291.970045414" Oct 30 06:32:05.557062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:05.557030 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:07.424162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:07.424130 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:09.154498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:09.154471 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:09.415427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:09.415353 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:32:10.925657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:10.925619 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:11.201514 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:11.201403 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt"] Oct 30 06:32:11.201708 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:11.201615 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" podUID="b65f0e55-4e85-4f37-a572-29d7e6635fec" containerName="istio-proxy" containerID="cri-o://4578f7edb272ea8dfdfbcb11154caa3d53ed846668d6a1939e131cc38a977734" gracePeriod=30 Oct 30 06:32:11.429786 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:11.429761 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:32:12.690574 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:12.690542 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:14.432137 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:14.432102 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:16.227870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.227831 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:16.484369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.484291 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:32:16.491596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.491462 6999 generic.go:356] "Generic (PLEG): container finished" podID="b65f0e55-4e85-4f37-a572-29d7e6635fec" containerID="4578f7edb272ea8dfdfbcb11154caa3d53ed846668d6a1939e131cc38a977734" exitCode=0 Oct 30 06:32:16.491596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.491504 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" event={"ID":"b65f0e55-4e85-4f37-a572-29d7e6635fec","Type":"ContainerDied","Data":"4578f7edb272ea8dfdfbcb11154caa3d53ed846668d6a1939e131cc38a977734"} Oct 30 06:32:16.491596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.491538 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" Oct 30 06:32:16.491596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.491549 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt" event={"ID":"b65f0e55-4e85-4f37-a572-29d7e6635fec","Type":"ContainerDied","Data":"6e8c9e92466c73edddf2ae93398ac49327e434121b8005b1462c54e3f9fb2e2a"} Oct 30 06:32:16.491596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.491571 6999 scope.go:122] "RemoveContainer" containerID="4578f7edb272ea8dfdfbcb11154caa3d53ed846668d6a1939e131cc38a977734" Oct 30 06:32:16.501588 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.501571 6999 scope.go:122] "RemoveContainer" containerID="4578f7edb272ea8dfdfbcb11154caa3d53ed846668d6a1939e131cc38a977734" Oct 30 06:32:16.501852 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:32:16.501835 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4578f7edb272ea8dfdfbcb11154caa3d53ed846668d6a1939e131cc38a977734\": container with ID starting with 4578f7edb272ea8dfdfbcb11154caa3d53ed846668d6a1939e131cc38a977734 not found: ID does not exist" containerID="4578f7edb272ea8dfdfbcb11154caa3d53ed846668d6a1939e131cc38a977734" Oct 30 06:32:16.501899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.501861 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4578f7edb272ea8dfdfbcb11154caa3d53ed846668d6a1939e131cc38a977734"} err="failed to get container status \"4578f7edb272ea8dfdfbcb11154caa3d53ed846668d6a1939e131cc38a977734\": rpc error: code = NotFound desc = could not find container \"4578f7edb272ea8dfdfbcb11154caa3d53ed846668d6a1939e131cc38a977734\": container with ID starting with 4578f7edb272ea8dfdfbcb11154caa3d53ed846668d6a1939e131cc38a977734 not found: ID does not exist" Oct 30 06:32:16.596935 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.596899 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65f0e55-4e85-4f37-a572-29d7e6635fec-kube-api-access-7nwww\" (UniqueName: \"kubernetes.io/projected/b65f0e55-4e85-4f37-a572-29d7e6635fec-kube-api-access-7nwww\") pod \"b65f0e55-4e85-4f37-a572-29d7e6635fec\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " Oct 30 06:32:16.597117 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.596977 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-podinfo\") pod \"b65f0e55-4e85-4f37-a572-29d7e6635fec\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " Oct 30 06:32:16.597117 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.597016 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-workload-certs\") pod \"b65f0e55-4e85-4f37-a572-29d7e6635fec\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " Oct 30 06:32:16.597295 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.597166 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b65f0e55-4e85-4f37-a572-29d7e6635fec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65f0e55-4e85-4f37-a572-29d7e6635fec-istiod-ca-cert\") pod \"b65f0e55-4e85-4f37-a572-29d7e6635fec\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " Oct 30 06:32:16.597358 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.597295 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-envoy\") pod \"b65f0e55-4e85-4f37-a572-29d7e6635fec\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " Oct 30 06:32:16.597358 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.597327 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-credential-socket\") pod \"b65f0e55-4e85-4f37-a572-29d7e6635fec\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " Oct 30 06:32:16.597358 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.597338 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-workload-certs" pod "b65f0e55-4e85-4f37-a572-29d7e6635fec" (UID: "b65f0e55-4e85-4f37-a572-29d7e6635fec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:16.597517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.597360 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-data\") pod \"b65f0e55-4e85-4f37-a572-29d7e6635fec\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " Oct 30 06:32:16.597517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.597390 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-workload-socket\") pod \"b65f0e55-4e85-4f37-a572-29d7e6635fec\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " Oct 30 06:32:16.597517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.597461 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-token\" (UniqueName: \"kubernetes.io/projected/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-token\") pod \"b65f0e55-4e85-4f37-a572-29d7e6635fec\" (UID: \"b65f0e55-4e85-4f37-a572-29d7e6635fec\") " Oct 30 06:32:16.597656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.597604 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-credential-socket" pod "b65f0e55-4e85-4f37-a572-29d7e6635fec" (UID: "b65f0e55-4e85-4f37-a572-29d7e6635fec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:16.597656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.597620 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-data" pod "b65f0e55-4e85-4f37-a572-29d7e6635fec" (UID: "b65f0e55-4e85-4f37-a572-29d7e6635fec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:16.597656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.597636 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b65f0e55-4e85-4f37-a572-29d7e6635fec-istiod-ca-cert" pod "b65f0e55-4e85-4f37-a572-29d7e6635fec" (UID: "b65f0e55-4e85-4f37-a572-29d7e6635fec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:32:16.597819 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.597787 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-workload-socket" pod "b65f0e55-4e85-4f37-a572-29d7e6635fec" (UID: "b65f0e55-4e85-4f37-a572-29d7e6635fec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:16.598040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.598018 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:16.598040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.598041 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b65f0e55-4e85-4f37-a572-29d7e6635fec-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:16.598197 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.598053 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:16.598197 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.598067 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:16.598197 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.598079 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:16.599405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.599378 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65f0e55-4e85-4f37-a572-29d7e6635fec-kube-api-access-7nwww" pod "b65f0e55-4e85-4f37-a572-29d7e6635fec" (UID: "b65f0e55-4e85-4f37-a572-29d7e6635fec"). InnerVolumeSpecName "kube-api-access-7nwww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:16.599738 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.599717 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-token" pod "b65f0e55-4e85-4f37-a572-29d7e6635fec" (UID: "b65f0e55-4e85-4f37-a572-29d7e6635fec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:16.599913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.599897 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-podinfo" pod "b65f0e55-4e85-4f37-a572-29d7e6635fec" (UID: "b65f0e55-4e85-4f37-a572-29d7e6635fec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:32:16.600020 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.599997 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-envoy" pod "b65f0e55-4e85-4f37-a572-29d7e6635fec" (UID: "b65f0e55-4e85-4f37-a572-29d7e6635fec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:16.699556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.699524 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:16.699556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.699551 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:16.699556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.699560 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nwww\" (UniqueName: \"kubernetes.io/projected/b65f0e55-4e85-4f37-a572-29d7e6635fec-kube-api-access-7nwww\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:16.699812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.699570 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b65f0e55-4e85-4f37-a572-29d7e6635fec-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:16.838972 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.838933 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt"] Oct 30 06:32:16.851000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:16.850960 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-wl5qt"] Oct 30 06:32:17.610070 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:17.610035 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b65f0e55-4e85-4f37-a572-29d7e6635fec" path="/var/lib/kubelet/pods/b65f0e55-4e85-4f37-a572-29d7e6635fec/volumes" Oct 30 06:32:18.025579 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:18.025483 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:19.816544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:19.816513 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:21.738448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:21.738412 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:23.541515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:23.541481 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:25.422116 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:25.422051 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:27.234462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:27.234429 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:28.478336 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:28.478305 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj"] Oct 30 06:32:28.549195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:28.549161 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:28.913969 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:28.913931 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn"] Oct 30 06:32:28.962641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:28.962601 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:29.162431 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:29.162294 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:31.032635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:31.032601 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:32.828572 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:32.828535 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:34.147076 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:32:34.147036 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:32:34.319898 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:34.319866 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:34.504956 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:34.504876 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x"] Oct 30 06:32:34.584798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:34.584757 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:34.856904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:34.856820 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:35.834561 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:35.834525 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:36.320802 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:36.320764 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:36.472965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:36.472900 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:36.826483 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:36.826444 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:37.749962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:37.749933 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:37.902916 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:37.902880 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:37.977980 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:37.977944 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5"] Oct 30 06:32:37.978276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:37.978239 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" podUID="e17ee302-1c6d-4e12-af38-41ba781e7379" containerName="istio-proxy" containerID="cri-o://d579a78c7b5ab0f1e409ce9324ca76e318284ed465cbae252f1c21b5598e92c9" gracePeriod=30 Oct 30 06:32:37.990306 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:37.990201 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh"] Oct 30 06:32:37.990566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:37.990534 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" podUID="a106c2d8-c5a5-4e22-b75d-922be5d0231a" containerName="istio-proxy" containerID="cri-o://f6f5f2b72cf4408bf0832cbdbc7e82b60f2e62b71c70fd721466758fb090a29b" gracePeriod=30 Oct 30 06:32:38.627686 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:38.627656 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:39.217314 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:39.217278 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:39.380502 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:39.380471 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:39.577926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:39.577825 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:39.738897 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:39.738865 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:39.960490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:39.960461 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:40.111240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:40.110607 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:40.441132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:40.441096 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:40.603643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:40.603611 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:40.751028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:40.750941 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:40.905770 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:40.905738 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:42.277802 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:42.277754 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:42.446512 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:42.446479 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:42.882983 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:42.882942 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:43.300597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.300571 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:32:43.322501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.322471 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:32:43.489946 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.489858 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-envoy\") pod \"e17ee302-1c6d-4e12-af38-41ba781e7379\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " Oct 30 06:32:43.490127 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.489943 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-podinfo\") pod \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " Oct 30 06:32:43.490127 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.489980 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-token\" (UniqueName: \"kubernetes.io/projected/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-token\") pod \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " Oct 30 06:32:43.490127 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.490076 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17ee302-1c6d-4e12-af38-41ba781e7379-kube-api-access-gpjpl\" (UniqueName: \"kubernetes.io/projected/e17ee302-1c6d-4e12-af38-41ba781e7379-kube-api-access-gpjpl\") pod \"e17ee302-1c6d-4e12-af38-41ba781e7379\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " Oct 30 06:32:43.490127 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.490117 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-credential-socket\") pod \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " Oct 30 06:32:43.490385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.490148 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e17ee302-1c6d-4e12-af38-41ba781e7379-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17ee302-1c6d-4e12-af38-41ba781e7379-istiod-ca-cert\") pod \"e17ee302-1c6d-4e12-af38-41ba781e7379\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " Oct 30 06:32:43.490385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.490225 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-workload-certs\") pod \"e17ee302-1c6d-4e12-af38-41ba781e7379\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " Oct 30 06:32:43.490385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.490284 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-credential-socket\") pod \"e17ee302-1c6d-4e12-af38-41ba781e7379\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " Oct 30 06:32:43.490385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.490313 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-data\") pod \"e17ee302-1c6d-4e12-af38-41ba781e7379\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " Oct 30 06:32:43.490385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.490352 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-envoy\") pod \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " Oct 30 06:32:43.490653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.490494 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-workload-certs\") pod \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " Oct 30 06:32:43.490653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.490506 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-credential-socket" pod "a106c2d8-c5a5-4e22-b75d-922be5d0231a" (UID: "a106c2d8-c5a5-4e22-b75d-922be5d0231a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:43.490653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.490525 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-podinfo\") pod \"e17ee302-1c6d-4e12-af38-41ba781e7379\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " Oct 30 06:32:43.490904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.490764 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-token\" (UniqueName: \"kubernetes.io/projected/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-token\") pod \"e17ee302-1c6d-4e12-af38-41ba781e7379\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " Oct 30 06:32:43.490904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.490843 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-workload-socket\") pod \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " Oct 30 06:32:43.490904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.490879 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istiod-ca-cert\") pod \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " Oct 30 06:32:43.491134 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.490916 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-workload-socket\") pod \"e17ee302-1c6d-4e12-af38-41ba781e7379\" (UID: \"e17ee302-1c6d-4e12-af38-41ba781e7379\") " Oct 30 06:32:43.491134 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.491016 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-data\") pod \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " Oct 30 06:32:43.491134 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.491028 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-credential-socket" pod "e17ee302-1c6d-4e12-af38-41ba781e7379" (UID: "e17ee302-1c6d-4e12-af38-41ba781e7379"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:43.491134 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.491074 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a106c2d8-c5a5-4e22-b75d-922be5d0231a-kube-api-access-p5nf5\" (UniqueName: \"kubernetes.io/projected/a106c2d8-c5a5-4e22-b75d-922be5d0231a-kube-api-access-p5nf5\") pod \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\" (UID: \"a106c2d8-c5a5-4e22-b75d-922be5d0231a\") " Oct 30 06:32:43.491497 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.491411 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e17ee302-1c6d-4e12-af38-41ba781e7379-istiod-ca-cert" pod "e17ee302-1c6d-4e12-af38-41ba781e7379" (UID: "e17ee302-1c6d-4e12-af38-41ba781e7379"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:32:43.491596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.491518 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-workload-socket" pod "a106c2d8-c5a5-4e22-b75d-922be5d0231a" (UID: "a106c2d8-c5a5-4e22-b75d-922be5d0231a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:43.491709 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.491671 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-workload-certs" pod "e17ee302-1c6d-4e12-af38-41ba781e7379" (UID: "e17ee302-1c6d-4e12-af38-41ba781e7379"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:43.492009 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.491989 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.492009 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.491998 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istiod-ca-cert" pod "a106c2d8-c5a5-4e22-b75d-922be5d0231a" (UID: "a106c2d8-c5a5-4e22-b75d-922be5d0231a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:32:43.492433 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.492016 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.492433 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.492033 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e17ee302-1c6d-4e12-af38-41ba781e7379-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.492433 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.492048 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.492433 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.492061 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.492433 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.492253 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-workload-socket" pod "e17ee302-1c6d-4e12-af38-41ba781e7379" (UID: "e17ee302-1c6d-4e12-af38-41ba781e7379"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:43.492433 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.492342 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-workload-certs" pod "a106c2d8-c5a5-4e22-b75d-922be5d0231a" (UID: "a106c2d8-c5a5-4e22-b75d-922be5d0231a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:43.492824 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.492465 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-data" pod "a106c2d8-c5a5-4e22-b75d-922be5d0231a" (UID: "a106c2d8-c5a5-4e22-b75d-922be5d0231a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:43.492824 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.492691 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-data" pod "e17ee302-1c6d-4e12-af38-41ba781e7379" (UID: "e17ee302-1c6d-4e12-af38-41ba781e7379"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:43.493196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.493171 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-token" pod "a106c2d8-c5a5-4e22-b75d-922be5d0231a" (UID: "a106c2d8-c5a5-4e22-b75d-922be5d0231a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:43.494202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.494156 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-envoy" pod "a106c2d8-c5a5-4e22-b75d-922be5d0231a" (UID: "a106c2d8-c5a5-4e22-b75d-922be5d0231a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:43.494531 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.494510 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-podinfo" pod "a106c2d8-c5a5-4e22-b75d-922be5d0231a" (UID: "a106c2d8-c5a5-4e22-b75d-922be5d0231a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:32:43.494863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.494839 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17ee302-1c6d-4e12-af38-41ba781e7379-kube-api-access-gpjpl" pod "e17ee302-1c6d-4e12-af38-41ba781e7379" (UID: "e17ee302-1c6d-4e12-af38-41ba781e7379"). InnerVolumeSpecName "kube-api-access-gpjpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:43.494966 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.494946 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-envoy" pod "e17ee302-1c6d-4e12-af38-41ba781e7379" (UID: "e17ee302-1c6d-4e12-af38-41ba781e7379"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:43.495054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.494966 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a106c2d8-c5a5-4e22-b75d-922be5d0231a-kube-api-access-p5nf5" pod "a106c2d8-c5a5-4e22-b75d-922be5d0231a" (UID: "a106c2d8-c5a5-4e22-b75d-922be5d0231a"). InnerVolumeSpecName "kube-api-access-p5nf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:43.495945 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.495915 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-token" pod "e17ee302-1c6d-4e12-af38-41ba781e7379" (UID: "e17ee302-1c6d-4e12-af38-41ba781e7379"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:43.496392 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.496367 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-podinfo" pod "e17ee302-1c6d-4e12-af38-41ba781e7379" (UID: "e17ee302-1c6d-4e12-af38-41ba781e7379"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:32:43.501478 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.501447 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" podUID="b3d4a9bc-8241-4e6b-8af0-2ada17fffceb" containerName="main" containerID="cri-o://3d464df6b8e2ef581182415edc30277ff03545f27d67788636771813e5b70b7d" gracePeriod=45 Oct 30 06:32:43.593789 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.593759 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.593789 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.593788 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.594069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.593809 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.594069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.593819 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.594069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.593829 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.594069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.593836 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.594069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.593845 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5nf5\" (UniqueName: \"kubernetes.io/projected/a106c2d8-c5a5-4e22-b75d-922be5d0231a-kube-api-access-p5nf5\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.594069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.593853 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.594069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.593862 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.594069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.593876 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.594069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.593889 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpjpl\" (UniqueName: \"kubernetes.io/projected/e17ee302-1c6d-4e12-af38-41ba781e7379-kube-api-access-gpjpl\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.594069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.593903 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e17ee302-1c6d-4e12-af38-41ba781e7379-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.594069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.593912 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a106c2d8-c5a5-4e22-b75d-922be5d0231a-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.720804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.720770 6999 generic.go:356] "Generic (PLEG): container finished" podID="a106c2d8-c5a5-4e22-b75d-922be5d0231a" containerID="f6f5f2b72cf4408bf0832cbdbc7e82b60f2e62b71c70fd721466758fb090a29b" exitCode=0 Oct 30 06:32:43.720987 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.720889 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" Oct 30 06:32:43.720987 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.720909 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" event={"ID":"a106c2d8-c5a5-4e22-b75d-922be5d0231a","Type":"ContainerDied","Data":"f6f5f2b72cf4408bf0832cbdbc7e82b60f2e62b71c70fd721466758fb090a29b"} Oct 30 06:32:43.720987 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.720970 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh" event={"ID":"a106c2d8-c5a5-4e22-b75d-922be5d0231a","Type":"ContainerDied","Data":"b579b3d26a1abaddee98844c678a3acc8a6d2deb1378a1befc2ba6a22375f648"} Oct 30 06:32:43.721158 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.720998 6999 scope.go:122] "RemoveContainer" containerID="f6f5f2b72cf4408bf0832cbdbc7e82b60f2e62b71c70fd721466758fb090a29b" Oct 30 06:32:43.726410 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.723300 6999 generic.go:356] "Generic (PLEG): container finished" podID="b3d4a9bc-8241-4e6b-8af0-2ada17fffceb" containerID="3d464df6b8e2ef581182415edc30277ff03545f27d67788636771813e5b70b7d" exitCode=0 Oct 30 06:32:43.726410 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.723444 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" event={"ID":"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb","Type":"ContainerDied","Data":"3d464df6b8e2ef581182415edc30277ff03545f27d67788636771813e5b70b7d"} Oct 30 06:32:43.727302 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.727279 6999 generic.go:356] "Generic (PLEG): container finished" podID="e17ee302-1c6d-4e12-af38-41ba781e7379" containerID="d579a78c7b5ab0f1e409ce9324ca76e318284ed465cbae252f1c21b5598e92c9" exitCode=0 Oct 30 06:32:43.727424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.727320 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" event={"ID":"e17ee302-1c6d-4e12-af38-41ba781e7379","Type":"ContainerDied","Data":"d579a78c7b5ab0f1e409ce9324ca76e318284ed465cbae252f1c21b5598e92c9"} Oct 30 06:32:43.727424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.727343 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" Oct 30 06:32:43.727424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.727358 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5" event={"ID":"e17ee302-1c6d-4e12-af38-41ba781e7379","Type":"ContainerDied","Data":"2d057154024c9aff9f2137c7615d5baaae142d6ac58a171dc069b68adbded14d"} Oct 30 06:32:43.732910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.732891 6999 scope.go:122] "RemoveContainer" containerID="f6f5f2b72cf4408bf0832cbdbc7e82b60f2e62b71c70fd721466758fb090a29b" Oct 30 06:32:43.733169 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:32:43.733148 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6f5f2b72cf4408bf0832cbdbc7e82b60f2e62b71c70fd721466758fb090a29b\": container with ID starting with f6f5f2b72cf4408bf0832cbdbc7e82b60f2e62b71c70fd721466758fb090a29b not found: ID does not exist" containerID="f6f5f2b72cf4408bf0832cbdbc7e82b60f2e62b71c70fd721466758fb090a29b" Oct 30 06:32:43.733236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.733178 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6f5f2b72cf4408bf0832cbdbc7e82b60f2e62b71c70fd721466758fb090a29b"} err="failed to get container status \"f6f5f2b72cf4408bf0832cbdbc7e82b60f2e62b71c70fd721466758fb090a29b\": rpc error: code = NotFound desc = could not find container \"f6f5f2b72cf4408bf0832cbdbc7e82b60f2e62b71c70fd721466758fb090a29b\": container with ID starting with f6f5f2b72cf4408bf0832cbdbc7e82b60f2e62b71c70fd721466758fb090a29b not found: ID does not exist" Oct 30 06:32:43.733236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.733196 6999 scope.go:122] "RemoveContainer" containerID="d579a78c7b5ab0f1e409ce9324ca76e318284ed465cbae252f1c21b5598e92c9" Oct 30 06:32:43.743151 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.743129 6999 scope.go:122] "RemoveContainer" containerID="d579a78c7b5ab0f1e409ce9324ca76e318284ed465cbae252f1c21b5598e92c9" Oct 30 06:32:43.743600 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:32:43.743576 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d579a78c7b5ab0f1e409ce9324ca76e318284ed465cbae252f1c21b5598e92c9\": container with ID starting with d579a78c7b5ab0f1e409ce9324ca76e318284ed465cbae252f1c21b5598e92c9 not found: ID does not exist" containerID="d579a78c7b5ab0f1e409ce9324ca76e318284ed465cbae252f1c21b5598e92c9" Oct 30 06:32:43.743679 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.743608 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d579a78c7b5ab0f1e409ce9324ca76e318284ed465cbae252f1c21b5598e92c9"} err="failed to get container status \"d579a78c7b5ab0f1e409ce9324ca76e318284ed465cbae252f1c21b5598e92c9\": rpc error: code = NotFound desc = could not find container \"d579a78c7b5ab0f1e409ce9324ca76e318284ed465cbae252f1c21b5598e92c9\": container with ID starting with d579a78c7b5ab0f1e409ce9324ca76e318284ed465cbae252f1c21b5598e92c9 not found: ID does not exist" Oct 30 06:32:43.757050 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.757031 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:32:43.856756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.856729 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh"] Oct 30 06:32:43.887099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.887067 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-cbvkh"] Oct 30 06:32:43.900125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.900096 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-tls-certs\") pod \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " Oct 30 06:32:43.900288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.900184 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-home\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-home\") pod \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " Oct 30 06:32:43.900288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.900273 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-tmp-dir\") pod \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " Oct 30 06:32:43.900422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.900352 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-model-cache\") pod \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " Oct 30 06:32:43.900482 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.900452 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-dshm\") pod \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " Oct 30 06:32:43.900482 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.900460 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-home" pod "b3d4a9bc-8241-4e6b-8af0-2ada17fffceb" (UID: "b3d4a9bc-8241-4e6b-8af0-2ada17fffceb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:43.900569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.900505 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-kube-api-access-8zxd4\" (UniqueName: \"kubernetes.io/projected/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-kube-api-access-8zxd4\") pod \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\" (UID: \"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb\") " Oct 30 06:32:43.900631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.900559 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-tmp-dir" pod "b3d4a9bc-8241-4e6b-8af0-2ada17fffceb" (UID: "b3d4a9bc-8241-4e6b-8af0-2ada17fffceb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:43.900631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.900593 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-model-cache" pod "b3d4a9bc-8241-4e6b-8af0-2ada17fffceb" (UID: "b3d4a9bc-8241-4e6b-8af0-2ada17fffceb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:43.900991 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.900969 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.901268 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.900996 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.901268 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.901010 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:43.902654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.902633 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-tls-certs" pod "b3d4a9bc-8241-4e6b-8af0-2ada17fffceb" (UID: "b3d4a9bc-8241-4e6b-8af0-2ada17fffceb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:32:43.902729 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.902656 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-dshm" pod "b3d4a9bc-8241-4e6b-8af0-2ada17fffceb" (UID: "b3d4a9bc-8241-4e6b-8af0-2ada17fffceb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:43.902729 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.902686 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-kube-api-access-8zxd4" pod "b3d4a9bc-8241-4e6b-8af0-2ada17fffceb" (UID: "b3d4a9bc-8241-4e6b-8af0-2ada17fffceb"). InnerVolumeSpecName "kube-api-access-8zxd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:43.922818 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.922792 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5"] Oct 30 06:32:43.929935 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.929912 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-9slg5"] Oct 30 06:32:43.936560 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.936532 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" podUID="dd88e3b9-268f-4bc4-ad34-cd295e804e81" containerName="main" containerID="cri-o://05d16d2930743567812e9ea7cbff211b88bafdebd706d520c9fe60b19e8834be" gracePeriod=45 Oct 30 06:32:43.966529 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:43.966477 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" podUID="dd88e3b9-268f-4bc4-ad34-cd295e804e81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.156:8000/ready\": dial tcp 10.134.0.156:8000: connect: connection refused" Oct 30 06:32:44.002365 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.002290 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:44.002365 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.002320 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:44.002365 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.002334 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zxd4\" (UniqueName: \"kubernetes.io/projected/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb-kube-api-access-8zxd4\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:44.238860 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.238834 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:32:44.307374 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.307290 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd88e3b9-268f-4bc4-ad34-cd295e804e81-tls-certs\" (UniqueName: \"kubernetes.io/secret/dd88e3b9-268f-4bc4-ad34-cd295e804e81-tls-certs\") pod \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " Oct 30 06:32:44.307854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.307384 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd88e3b9-268f-4bc4-ad34-cd295e804e81-kube-api-access-6dz6p\" (UniqueName: \"kubernetes.io/projected/dd88e3b9-268f-4bc4-ad34-cd295e804e81-kube-api-access-6dz6p\") pod \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " Oct 30 06:32:44.307854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.307423 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-home\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-home\") pod \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " Oct 30 06:32:44.307854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.307496 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-model-cache\") pod \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " Oct 30 06:32:44.307854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.307620 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-tmp-dir\") pod \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " Oct 30 06:32:44.307854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.307695 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-dshm\") pod \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\" (UID: \"dd88e3b9-268f-4bc4-ad34-cd295e804e81\") " Oct 30 06:32:44.308341 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.307870 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-tmp-dir" pod "dd88e3b9-268f-4bc4-ad34-cd295e804e81" (UID: "dd88e3b9-268f-4bc4-ad34-cd295e804e81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:44.308341 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.307894 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-model-cache" pod "dd88e3b9-268f-4bc4-ad34-cd295e804e81" (UID: "dd88e3b9-268f-4bc4-ad34-cd295e804e81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:44.308341 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.308293 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:44.308341 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.308318 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:44.308555 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.308537 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-home" pod "dd88e3b9-268f-4bc4-ad34-cd295e804e81" (UID: "dd88e3b9-268f-4bc4-ad34-cd295e804e81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:44.310509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.310484 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd88e3b9-268f-4bc4-ad34-cd295e804e81-kube-api-access-6dz6p" pod "dd88e3b9-268f-4bc4-ad34-cd295e804e81" (UID: "dd88e3b9-268f-4bc4-ad34-cd295e804e81"). InnerVolumeSpecName "kube-api-access-6dz6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:44.310631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.310526 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd88e3b9-268f-4bc4-ad34-cd295e804e81-tls-certs" pod "dd88e3b9-268f-4bc4-ad34-cd295e804e81" (UID: "dd88e3b9-268f-4bc4-ad34-cd295e804e81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:32:44.310631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.310618 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-dshm" pod "dd88e3b9-268f-4bc4-ad34-cd295e804e81" (UID: "dd88e3b9-268f-4bc4-ad34-cd295e804e81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:44.421243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.410851 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dz6p\" (UniqueName: \"kubernetes.io/projected/dd88e3b9-268f-4bc4-ad34-cd295e804e81-kube-api-access-6dz6p\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:44.421243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.410888 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:44.421243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.410903 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dd88e3b9-268f-4bc4-ad34-cd295e804e81-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:44.421243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.410917 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd88e3b9-268f-4bc4-ad34-cd295e804e81-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:44.736645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.736606 6999 generic.go:356] "Generic (PLEG): container finished" podID="dd88e3b9-268f-4bc4-ad34-cd295e804e81" containerID="05d16d2930743567812e9ea7cbff211b88bafdebd706d520c9fe60b19e8834be" exitCode=0 Oct 30 06:32:44.736847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.736672 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" Oct 30 06:32:44.736847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.736695 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" event={"ID":"dd88e3b9-268f-4bc4-ad34-cd295e804e81","Type":"ContainerDied","Data":"05d16d2930743567812e9ea7cbff211b88bafdebd706d520c9fe60b19e8834be"} Oct 30 06:32:44.736847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.736741 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn" event={"ID":"dd88e3b9-268f-4bc4-ad34-cd295e804e81","Type":"ContainerDied","Data":"f4309ad7b319270e5d58fceb746afa668b0a67ccc775596fdce5b53972abc9dc"} Oct 30 06:32:44.736847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.736759 6999 scope.go:122] "RemoveContainer" containerID="05d16d2930743567812e9ea7cbff211b88bafdebd706d520c9fe60b19e8834be" Oct 30 06:32:44.747193 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.747134 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" event={"ID":"b3d4a9bc-8241-4e6b-8af0-2ada17fffceb","Type":"ContainerDied","Data":"adc1e813565c861778314bb5ad51a2bbe790e302ffdbd7aa48ac8ef1a18efae7"} Oct 30 06:32:44.747193 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.747152 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj" Oct 30 06:32:44.754904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.754886 6999 scope.go:122] "RemoveContainer" containerID="05d16d2930743567812e9ea7cbff211b88bafdebd706d520c9fe60b19e8834be" Oct 30 06:32:44.755186 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:32:44.755169 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d16d2930743567812e9ea7cbff211b88bafdebd706d520c9fe60b19e8834be\": container with ID starting with 05d16d2930743567812e9ea7cbff211b88bafdebd706d520c9fe60b19e8834be not found: ID does not exist" containerID="05d16d2930743567812e9ea7cbff211b88bafdebd706d520c9fe60b19e8834be" Oct 30 06:32:44.755267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.755195 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d16d2930743567812e9ea7cbff211b88bafdebd706d520c9fe60b19e8834be"} err="failed to get container status \"05d16d2930743567812e9ea7cbff211b88bafdebd706d520c9fe60b19e8834be\": rpc error: code = NotFound desc = could not find container \"05d16d2930743567812e9ea7cbff211b88bafdebd706d520c9fe60b19e8834be\": container with ID starting with 05d16d2930743567812e9ea7cbff211b88bafdebd706d520c9fe60b19e8834be not found: ID does not exist" Oct 30 06:32:44.755267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.755238 6999 scope.go:122] "RemoveContainer" containerID="3d464df6b8e2ef581182415edc30277ff03545f27d67788636771813e5b70b7d" Oct 30 06:32:44.814358 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.814331 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn"] Oct 30 06:32:44.823070 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.822530 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6e2548/auth-test-model-6e2548-kserve-64478d5445-7njnn"] Oct 30 06:32:44.868041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.868007 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj"] Oct 30 06:32:44.876332 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.876299 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:44.877103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:44.877084 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-645f2b/auth-test-model-645f2b-kserve-7f57c97fdf-5gmfj"] Oct 30 06:32:45.055555 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:45.055454 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:45.254038 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:45.254009 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:45.447489 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:45.447460 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:45.623231 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:45.623180 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a106c2d8-c5a5-4e22-b75d-922be5d0231a" path="/var/lib/kubelet/pods/a106c2d8-c5a5-4e22-b75d-922be5d0231a/volumes" Oct 30 06:32:45.623754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:45.623731 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3d4a9bc-8241-4e6b-8af0-2ada17fffceb" path="/var/lib/kubelet/pods/b3d4a9bc-8241-4e6b-8af0-2ada17fffceb/volumes" Oct 30 06:32:45.624203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:45.624185 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd88e3b9-268f-4bc4-ad34-cd295e804e81" path="/var/lib/kubelet/pods/dd88e3b9-268f-4bc4-ad34-cd295e804e81/volumes" Oct 30 06:32:45.625006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:45.624985 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e17ee302-1c6d-4e12-af38-41ba781e7379" path="/var/lib/kubelet/pods/e17ee302-1c6d-4e12-af38-41ba781e7379/volumes" Oct 30 06:32:46.598383 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.598347 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:46.739410 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.739376 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns"] Oct 30 06:32:46.739949 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.739936 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b3d4a9bc-8241-4e6b-8af0-2ada17fffceb" containerName="main" Oct 30 06:32:46.740012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.739951 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d4a9bc-8241-4e6b-8af0-2ada17fffceb" containerName="main" Oct 30 06:32:46.740012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.739969 6999 cpu_manager.go:414] "containerMap: removing container" podUID="dd88e3b9-268f-4bc4-ad34-cd295e804e81" containerName="main" Oct 30 06:32:46.740012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.739974 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd88e3b9-268f-4bc4-ad34-cd295e804e81" containerName="main" Oct 30 06:32:46.740012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.739985 6999 cpu_manager.go:414] "containerMap: removing container" podUID="a106c2d8-c5a5-4e22-b75d-922be5d0231a" containerName="istio-proxy" Oct 30 06:32:46.740012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.739990 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a106c2d8-c5a5-4e22-b75d-922be5d0231a" containerName="istio-proxy" Oct 30 06:32:46.740012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.739999 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b65f0e55-4e85-4f37-a572-29d7e6635fec" containerName="istio-proxy" Oct 30 06:32:46.740012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.740004 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b65f0e55-4e85-4f37-a572-29d7e6635fec" containerName="istio-proxy" Oct 30 06:32:46.740263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.740018 6999 cpu_manager.go:414] "containerMap: removing container" podUID="e17ee302-1c6d-4e12-af38-41ba781e7379" containerName="istio-proxy" Oct 30 06:32:46.740263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.740023 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e17ee302-1c6d-4e12-af38-41ba781e7379" containerName="istio-proxy" Oct 30 06:32:46.740263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.740099 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d4a9bc-8241-4e6b-8af0-2ada17fffceb" containerName="main" Oct 30 06:32:46.740263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.740109 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd88e3b9-268f-4bc4-ad34-cd295e804e81" containerName="main" Oct 30 06:32:46.740263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.740117 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="a106c2d8-c5a5-4e22-b75d-922be5d0231a" containerName="istio-proxy" Oct 30 06:32:46.740263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.740123 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b65f0e55-4e85-4f37-a572-29d7e6635fec" containerName="istio-proxy" Oct 30 06:32:46.740263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.740130 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="e17ee302-1c6d-4e12-af38-41ba781e7379" containerName="istio-proxy" Oct 30 06:32:46.745127 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.745108 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" Oct 30 06:32:46.754417 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.754383 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:46.774601 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.774576 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-ad1661-dockercfg-wz2s9\"" Oct 30 06:32:46.774601 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.774577 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-ad1661\"" Oct 30 06:32:46.777395 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.777374 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns"] Oct 30 06:32:46.840785 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.840735 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2xr\" (UniqueName: \"kubernetes.io/projected/ded3226b-1e88-48ff-a599-268f411abc76-kube-api-access-4z2xr\") pod \"payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns\" (UID: \"ded3226b-1e88-48ff-a599-268f411abc76\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" Oct 30 06:32:46.841005 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.840911 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded3226b-1e88-48ff-a599-268f411abc76-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns\" (UID: \"ded3226b-1e88-48ff-a599-268f411abc76\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" Oct 30 06:32:46.885728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.885697 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8"] Oct 30 06:32:46.890120 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.890093 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" Oct 30 06:32:46.939538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.939505 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8"] Oct 30 06:32:46.942699 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.942670 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2xr\" (UniqueName: \"kubernetes.io/projected/ded3226b-1e88-48ff-a599-268f411abc76-kube-api-access-4z2xr\") pod \"payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns\" (UID: \"ded3226b-1e88-48ff-a599-268f411abc76\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" Oct 30 06:32:46.942917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.942750 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62ce8a0-4a42-4943-8ced-5f033661b9be-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8\" (UID: \"e62ce8a0-4a42-4943-8ced-5f033661b9be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" Oct 30 06:32:46.942917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.942783 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjnds\" (UniqueName: \"kubernetes.io/projected/e62ce8a0-4a42-4943-8ced-5f033661b9be-kube-api-access-zjnds\") pod \"payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8\" (UID: \"e62ce8a0-4a42-4943-8ced-5f033661b9be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" Oct 30 06:32:46.943046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.942924 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded3226b-1e88-48ff-a599-268f411abc76-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns\" (UID: \"ded3226b-1e88-48ff-a599-268f411abc76\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" Oct 30 06:32:46.943791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.943762 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded3226b-1e88-48ff-a599-268f411abc76-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns\" (UID: \"ded3226b-1e88-48ff-a599-268f411abc76\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" Oct 30 06:32:46.988342 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:46.988310 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2xr\" (UniqueName: \"kubernetes.io/projected/ded3226b-1e88-48ff-a599-268f411abc76-kube-api-access-4z2xr\") pod \"payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns\" (UID: \"ded3226b-1e88-48ff-a599-268f411abc76\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" Oct 30 06:32:47.044382 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.044346 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62ce8a0-4a42-4943-8ced-5f033661b9be-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8\" (UID: \"e62ce8a0-4a42-4943-8ced-5f033661b9be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" Oct 30 06:32:47.044541 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.044398 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjnds\" (UniqueName: \"kubernetes.io/projected/e62ce8a0-4a42-4943-8ced-5f033661b9be-kube-api-access-zjnds\") pod \"payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8\" (UID: \"e62ce8a0-4a42-4943-8ced-5f033661b9be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" Oct 30 06:32:47.045282 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.045254 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62ce8a0-4a42-4943-8ced-5f033661b9be-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8\" (UID: \"e62ce8a0-4a42-4943-8ced-5f033661b9be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" Oct 30 06:32:47.058429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.058394 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" Oct 30 06:32:47.068750 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.068721 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjnds\" (UniqueName: \"kubernetes.io/projected/e62ce8a0-4a42-4943-8ced-5f033661b9be-kube-api-access-zjnds\") pod \"payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8\" (UID: \"e62ce8a0-4a42-4943-8ced-5f033661b9be\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" Oct 30 06:32:47.209229 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.209126 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" Oct 30 06:32:47.384843 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.384807 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns"] Oct 30 06:32:47.386988 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:32:47.386960 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded3226b_1e88_48ff_a599_268f411abc76.slice/crio-98ddaeef3859ac5d8e11c140598d2b267b7fa393c18c3754ffb7f8526ca4e71b WatchSource:0}: Error finding container 98ddaeef3859ac5d8e11c140598d2b267b7fa393c18c3754ffb7f8526ca4e71b: Status 404 returned error can't find the container with id 98ddaeef3859ac5d8e11c140598d2b267b7fa393c18c3754ffb7f8526ca4e71b Oct 30 06:32:47.521957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.521933 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8"] Oct 30 06:32:47.522990 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:32:47.522968 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62ce8a0_4a42_4943_8ced_5f033661b9be.slice/crio-45761d82ffb46fb08a760c3febc41d18eede6e3a8da8b0b0eacfc45acbc58edf WatchSource:0}: Error finding container 45761d82ffb46fb08a760c3febc41d18eede6e3a8da8b0b0eacfc45acbc58edf: Status 404 returned error can't find the container with id 45761d82ffb46fb08a760c3febc41d18eede6e3a8da8b0b0eacfc45acbc58edf Oct 30 06:32:47.549609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.549565 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:47.774629 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.774536 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" event={"ID":"e62ce8a0-4a42-4943-8ced-5f033661b9be","Type":"ContainerStarted","Data":"039c75c6e16b0e87329ec47a45465a2953024ddc97b71ad7629a0d8d4bfb1564"} Oct 30 06:32:47.774629 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.774596 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" event={"ID":"e62ce8a0-4a42-4943-8ced-5f033661b9be","Type":"ContainerStarted","Data":"45761d82ffb46fb08a760c3febc41d18eede6e3a8da8b0b0eacfc45acbc58edf"} Oct 30 06:32:47.775146 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.774727 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" Oct 30 06:32:47.776204 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.776178 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" event={"ID":"ded3226b-1e88-48ff-a599-268f411abc76","Type":"ContainerStarted","Data":"bd9315522559fabb00bac898d751a9a89f11244bfe8374534741219fb4997f70"} Oct 30 06:32:47.776324 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.776252 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" Oct 30 06:32:47.776324 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.776274 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" event={"ID":"ded3226b-1e88-48ff-a599-268f411abc76","Type":"ContainerStarted","Data":"98ddaeef3859ac5d8e11c140598d2b267b7fa393c18c3754ffb7f8526ca4e71b"} Oct 30 06:32:47.822651 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.822596 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" podStartSLOduration=1.822577764 podStartE2EDuration="1.822577764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:47.820196895 +0000 UTC m=+2334.329216794" watchObservedRunningTime="2026-10-30 06:32:47.822577764 +0000 UTC m=+2334.331597655" Oct 30 06:32:47.862478 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:47.862421 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" podStartSLOduration=1.862407726 podStartE2EDuration="1.862407726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:47.859624329 +0000 UTC m=+2334.368644220" watchObservedRunningTime="2026-10-30 06:32:47.862407726 +0000 UTC m=+2334.371427671" Oct 30 06:32:48.526845 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.526808 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:48.667023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.666984 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:48.668025 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.667997 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9"] Oct 30 06:32:48.672190 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.672171 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" Oct 30 06:32:48.698775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.698742 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-a0083e-dockercfg-ngnnt\"" Oct 30 06:32:48.699000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.698796 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-a0083e\"" Oct 30 06:32:48.701661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.701638 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9"] Oct 30 06:32:48.763699 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.763661 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp"] Oct 30 06:32:48.766753 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.766719 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb73c18a-0b2c-44b4-a5de-133963b43a88-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9\" (UID: \"fb73c18a-0b2c-44b4-a5de-133963b43a88\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" Oct 30 06:32:48.767081 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.767052 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnl97\" (UniqueName: \"kubernetes.io/projected/fb73c18a-0b2c-44b4-a5de-133963b43a88-kube-api-access-lnl97\") pod \"payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9\" (UID: \"fb73c18a-0b2c-44b4-a5de-133963b43a88\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" Oct 30 06:32:48.774767 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.774739 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" Oct 30 06:32:48.794585 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.794553 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp"] Oct 30 06:32:48.846776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.846744 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7312-1a493a"] Oct 30 06:32:48.851435 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.851411 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-1a493a" Oct 30 06:32:48.868013 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.867983 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p6zmk\"" Oct 30 06:32:48.869139 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.869111 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8nht\" (UniqueName: \"kubernetes.io/projected/e4de3e31-9dff-43fc-8553-f859e41a1583-kube-api-access-r8nht\") pod \"payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp\" (UID: \"e4de3e31-9dff-43fc-8553-f859e41a1583\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" Oct 30 06:32:48.869286 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.869186 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4de3e31-9dff-43fc-8553-f859e41a1583-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp\" (UID: \"e4de3e31-9dff-43fc-8553-f859e41a1583\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" Oct 30 06:32:48.869286 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.869227 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb73c18a-0b2c-44b4-a5de-133963b43a88-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9\" (UID: \"fb73c18a-0b2c-44b4-a5de-133963b43a88\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" Oct 30 06:32:48.869394 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.869294 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgcmm\" (UniqueName: \"kubernetes.io/projected/c4050c60-cfb6-456f-bc1c-42851a8fbdd5-kube-api-access-xgcmm\") pod \"test-curl-7312-1a493a\" (UID: \"c4050c60-cfb6-456f-bc1c-42851a8fbdd5\") " pod="opendatahub/test-curl-7312-1a493a" Oct 30 06:32:48.869394 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.869347 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnl97\" (UniqueName: \"kubernetes.io/projected/fb73c18a-0b2c-44b4-a5de-133963b43a88-kube-api-access-lnl97\") pod \"payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9\" (UID: \"fb73c18a-0b2c-44b4-a5de-133963b43a88\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" Oct 30 06:32:48.870412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.870390 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7312-1a493a"] Oct 30 06:32:48.870753 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.870731 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb73c18a-0b2c-44b4-a5de-133963b43a88-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9\" (UID: \"fb73c18a-0b2c-44b4-a5de-133963b43a88\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" Oct 30 06:32:48.892948 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.892911 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnl97\" (UniqueName: \"kubernetes.io/projected/fb73c18a-0b2c-44b4-a5de-133963b43a88-kube-api-access-lnl97\") pod \"payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9\" (UID: \"fb73c18a-0b2c-44b4-a5de-133963b43a88\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" Oct 30 06:32:48.970287 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.970237 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8nht\" (UniqueName: \"kubernetes.io/projected/e4de3e31-9dff-43fc-8553-f859e41a1583-kube-api-access-r8nht\") pod \"payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp\" (UID: \"e4de3e31-9dff-43fc-8553-f859e41a1583\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" Oct 30 06:32:48.970494 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.970340 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4de3e31-9dff-43fc-8553-f859e41a1583-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp\" (UID: \"e4de3e31-9dff-43fc-8553-f859e41a1583\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" Oct 30 06:32:48.970494 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.970405 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgcmm\" (UniqueName: \"kubernetes.io/projected/c4050c60-cfb6-456f-bc1c-42851a8fbdd5-kube-api-access-xgcmm\") pod \"test-curl-7312-1a493a\" (UID: \"c4050c60-cfb6-456f-bc1c-42851a8fbdd5\") " pod="opendatahub/test-curl-7312-1a493a" Oct 30 06:32:48.971069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.971050 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4de3e31-9dff-43fc-8553-f859e41a1583-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp\" (UID: \"e4de3e31-9dff-43fc-8553-f859e41a1583\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" Oct 30 06:32:48.988399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.988367 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" Oct 30 06:32:48.993780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.993747 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgcmm\" (UniqueName: \"kubernetes.io/projected/c4050c60-cfb6-456f-bc1c-42851a8fbdd5-kube-api-access-xgcmm\") pod \"test-curl-7312-1a493a\" (UID: \"c4050c60-cfb6-456f-bc1c-42851a8fbdd5\") " pod="opendatahub/test-curl-7312-1a493a" Oct 30 06:32:48.993908 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:48.993893 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8nht\" (UniqueName: \"kubernetes.io/projected/e4de3e31-9dff-43fc-8553-f859e41a1583-kube-api-access-r8nht\") pod \"payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp\" (UID: \"e4de3e31-9dff-43fc-8553-f859e41a1583\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" Oct 30 06:32:49.099281 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.098897 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" Oct 30 06:32:49.167475 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.167449 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-1a493a" Oct 30 06:32:49.308386 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.308300 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9"] Oct 30 06:32:49.390791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.390762 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:49.414848 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.414820 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp"] Oct 30 06:32:49.416899 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:32:49.416874 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4de3e31_9dff_43fc_8553_f859e41a1583.slice/crio-3ae9c64acaf8c8f31d6b10fb3f4bea43ebe39fdb1f8b1fa4d11a41b11436aef8 WatchSource:0}: Error finding container 3ae9c64acaf8c8f31d6b10fb3f4bea43ebe39fdb1f8b1fa4d11a41b11436aef8: Status 404 returned error can't find the container with id 3ae9c64acaf8c8f31d6b10fb3f4bea43ebe39fdb1f8b1fa4d11a41b11436aef8 Oct 30 06:32:49.493911 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.493877 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7312-1a493a"] Oct 30 06:32:49.531699 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.531649 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" podUID="97ead49a-073d-4bf3-bd25-afef28d6c83e" containerName="main" containerID="cri-o://59c85b859c1cc5e36e80551f436fdc2f6df021fedab07a7ebb60fcb3531b5c08" gracePeriod=45 Oct 30 06:32:49.546094 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.546064 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:49.704812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.704774 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:49.795030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.794994 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" event={"ID":"e4de3e31-9dff-43fc-8553-f859e41a1583","Type":"ContainerStarted","Data":"ad42af138fcb0b7c94699ac5547e4d96395ed3755bc0b9247961cc8d284a31ff"} Oct 30 06:32:49.795567 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.795057 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" event={"ID":"e4de3e31-9dff-43fc-8553-f859e41a1583","Type":"ContainerStarted","Data":"3ae9c64acaf8c8f31d6b10fb3f4bea43ebe39fdb1f8b1fa4d11a41b11436aef8"} Oct 30 06:32:49.795567 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.795200 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" Oct 30 06:32:49.796806 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.796776 6999 generic.go:356] "Generic (PLEG): container finished" podID="97ead49a-073d-4bf3-bd25-afef28d6c83e" containerID="59c85b859c1cc5e36e80551f436fdc2f6df021fedab07a7ebb60fcb3531b5c08" exitCode=0 Oct 30 06:32:49.796933 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.796841 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" event={"ID":"97ead49a-073d-4bf3-bd25-afef28d6c83e","Type":"ContainerDied","Data":"59c85b859c1cc5e36e80551f436fdc2f6df021fedab07a7ebb60fcb3531b5c08"} Oct 30 06:32:49.798428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.798404 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" event={"ID":"fb73c18a-0b2c-44b4-a5de-133963b43a88","Type":"ContainerStarted","Data":"fa9e1114aec9b183c703ace40c88f72bc5f4f106aad545e4d8791f9543177c02"} Oct 30 06:32:49.798560 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.798444 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" event={"ID":"fb73c18a-0b2c-44b4-a5de-133963b43a88","Type":"ContainerStarted","Data":"02cb342309529ffbfd1160559736fa1758872644f29b67e19a1a622cc6b4d714"} Oct 30 06:32:49.798560 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.798465 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" Oct 30 06:32:49.800105 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.800084 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7312-1a493a" event={"ID":"c4050c60-cfb6-456f-bc1c-42851a8fbdd5","Type":"ContainerStarted","Data":"d8b6dfb90e15adbcfea893f6d78dad438276d2170654323a4256abf146639fc9"} Oct 30 06:32:49.800259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.800113 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7312-1a493a" event={"ID":"c4050c60-cfb6-456f-bc1c-42851a8fbdd5","Type":"ContainerStarted","Data":"078a1939ab31bc0c157621f0a25f9ebc0210b9fea15e27a48492246880a45e7e"} Oct 30 06:32:49.817655 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.817636 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:32:49.837050 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.837007 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" podStartSLOduration=1.836995068 podStartE2EDuration="1.836995068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:49.834660506 +0000 UTC m=+2336.343680392" watchObservedRunningTime="2026-10-30 06:32:49.836995068 +0000 UTC m=+2336.346014952" Oct 30 06:32:49.882463 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.882436 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97ead49a-073d-4bf3-bd25-afef28d6c83e-kube-api-access-nqz4g\" (UniqueName: \"kubernetes.io/projected/97ead49a-073d-4bf3-bd25-afef28d6c83e-kube-api-access-nqz4g\") pod \"97ead49a-073d-4bf3-bd25-afef28d6c83e\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " Oct 30 06:32:49.882776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.882510 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-dshm\") pod \"97ead49a-073d-4bf3-bd25-afef28d6c83e\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " Oct 30 06:32:49.882776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.882582 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-model-cache\") pod \"97ead49a-073d-4bf3-bd25-afef28d6c83e\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " Oct 30 06:32:49.882776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.882612 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97ead49a-073d-4bf3-bd25-afef28d6c83e-tls-certs\" (UniqueName: \"kubernetes.io/secret/97ead49a-073d-4bf3-bd25-afef28d6c83e-tls-certs\") pod \"97ead49a-073d-4bf3-bd25-afef28d6c83e\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " Oct 30 06:32:49.882776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.882732 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-home\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-home\") pod \"97ead49a-073d-4bf3-bd25-afef28d6c83e\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " Oct 30 06:32:49.883066 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.882879 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-tmp-dir\") pod \"97ead49a-073d-4bf3-bd25-afef28d6c83e\" (UID: \"97ead49a-073d-4bf3-bd25-afef28d6c83e\") " Oct 30 06:32:49.883266 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.883237 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-model-cache" pod "97ead49a-073d-4bf3-bd25-afef28d6c83e" (UID: "97ead49a-073d-4bf3-bd25-afef28d6c83e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:49.886810 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.883480 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-home" pod "97ead49a-073d-4bf3-bd25-afef28d6c83e" (UID: "97ead49a-073d-4bf3-bd25-afef28d6c83e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:49.886810 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.883722 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-tmp-dir" pod "97ead49a-073d-4bf3-bd25-afef28d6c83e" (UID: "97ead49a-073d-4bf3-bd25-afef28d6c83e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:49.886810 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.883916 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:49.886810 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.884181 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:49.887096 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.886995 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ead49a-073d-4bf3-bd25-afef28d6c83e-kube-api-access-nqz4g" pod "97ead49a-073d-4bf3-bd25-afef28d6c83e" (UID: "97ead49a-073d-4bf3-bd25-afef28d6c83e"). InnerVolumeSpecName "kube-api-access-nqz4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:49.887518 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.887493 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-dshm" pod "97ead49a-073d-4bf3-bd25-afef28d6c83e" (UID: "97ead49a-073d-4bf3-bd25-afef28d6c83e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:32:49.889307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.889283 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97ead49a-073d-4bf3-bd25-afef28d6c83e-tls-certs" pod "97ead49a-073d-4bf3-bd25-afef28d6c83e" (UID: "97ead49a-073d-4bf3-bd25-afef28d6c83e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:32:49.917734 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.917683 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" podStartSLOduration=1.917668334 podStartE2EDuration="1.917668334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:49.915910214 +0000 UTC m=+2336.424930132" watchObservedRunningTime="2026-10-30 06:32:49.917668334 +0000 UTC m=+2336.426688222" Oct 30 06:32:49.954618 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.954575 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7312-1a493a" podStartSLOduration=1.954562452 podStartE2EDuration="1.954562452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:49.952966065 +0000 UTC m=+2336.461985950" watchObservedRunningTime="2026-10-30 06:32:49.954562452 +0000 UTC m=+2336.463582338" Oct 30 06:32:49.987806 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.987772 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqz4g\" (UniqueName: \"kubernetes.io/projected/97ead49a-073d-4bf3-bd25-afef28d6c83e-kube-api-access-nqz4g\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:49.987806 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.987809 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:49.988056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.987825 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ead49a-073d-4bf3-bd25-afef28d6c83e-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:49.988056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:49.987840 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ead49a-073d-4bf3-bd25-afef28d6c83e-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:50.809500 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:50.809458 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" event={"ID":"97ead49a-073d-4bf3-bd25-afef28d6c83e","Type":"ContainerDied","Data":"116c12fc8cfc23a4adbf30df912c22c85a31b78e546d55db7f1ccb7ab480d3c6"} Oct 30 06:32:50.809500 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:50.809489 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x" Oct 30 06:32:50.810019 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:50.809524 6999 scope.go:122] "RemoveContainer" containerID="59c85b859c1cc5e36e80551f436fdc2f6df021fedab07a7ebb60fcb3531b5c08" Oct 30 06:32:50.860613 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:50.860582 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x"] Oct 30 06:32:50.873040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:50.873007 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-868b1f/auth-test-model-868b1f-kserve-6fc5bd4ccc-5jr9x"] Oct 30 06:32:50.876423 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:50.875706 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7312-1a493a"] Oct 30 06:32:50.878871 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:50.878844 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7312-1a493a"] Oct 30 06:32:51.417463 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:51.417423 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:51.609684 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:51.609651 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97ead49a-073d-4bf3-bd25-afef28d6c83e" path="/var/lib/kubelet/pods/97ead49a-073d-4bf3-bd25-afef28d6c83e/volumes" Oct 30 06:32:51.816884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:51.816799 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7312-1a493a" podUID="c4050c60-cfb6-456f-bc1c-42851a8fbdd5" containerName="test-curl-7312-1a493a" containerID="cri-o://d8b6dfb90e15adbcfea893f6d78dad438276d2170654323a4256abf146639fc9" gracePeriod=2 Oct 30 06:32:53.188946 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:53.188906 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:54.033561 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:54.033533 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-1a493a" Oct 30 06:32:54.140659 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:54.140628 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4050c60-cfb6-456f-bc1c-42851a8fbdd5-kube-api-access-xgcmm\" (UniqueName: \"kubernetes.io/projected/c4050c60-cfb6-456f-bc1c-42851a8fbdd5-kube-api-access-xgcmm\") pod \"c4050c60-cfb6-456f-bc1c-42851a8fbdd5\" (UID: \"c4050c60-cfb6-456f-bc1c-42851a8fbdd5\") " Oct 30 06:32:54.143255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:54.143188 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4050c60-cfb6-456f-bc1c-42851a8fbdd5-kube-api-access-xgcmm" pod "c4050c60-cfb6-456f-bc1c-42851a8fbdd5" (UID: "c4050c60-cfb6-456f-bc1c-42851a8fbdd5"). InnerVolumeSpecName "kube-api-access-xgcmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:54.242773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:54.242735 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgcmm\" (UniqueName: \"kubernetes.io/projected/c4050c60-cfb6-456f-bc1c-42851a8fbdd5-kube-api-access-xgcmm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:32:54.842463 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:54.842422 6999 generic.go:356] "Generic (PLEG): container finished" podID="c4050c60-cfb6-456f-bc1c-42851a8fbdd5" containerID="d8b6dfb90e15adbcfea893f6d78dad438276d2170654323a4256abf146639fc9" exitCode=137 Oct 30 06:32:54.842684 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:54.842541 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7312-1a493a" Oct 30 06:32:54.842684 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:54.842635 6999 scope.go:122] "RemoveContainer" containerID="d8b6dfb90e15adbcfea893f6d78dad438276d2170654323a4256abf146639fc9" Oct 30 06:32:54.853135 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:54.853105 6999 scope.go:122] "RemoveContainer" containerID="d8b6dfb90e15adbcfea893f6d78dad438276d2170654323a4256abf146639fc9" Oct 30 06:32:54.853393 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:32:54.853371 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b6dfb90e15adbcfea893f6d78dad438276d2170654323a4256abf146639fc9\": container with ID starting with d8b6dfb90e15adbcfea893f6d78dad438276d2170654323a4256abf146639fc9 not found: ID does not exist" containerID="d8b6dfb90e15adbcfea893f6d78dad438276d2170654323a4256abf146639fc9" Oct 30 06:32:54.853449 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:54.853402 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b6dfb90e15adbcfea893f6d78dad438276d2170654323a4256abf146639fc9"} err="failed to get container status \"d8b6dfb90e15adbcfea893f6d78dad438276d2170654323a4256abf146639fc9\": rpc error: code = NotFound desc = could not find container \"d8b6dfb90e15adbcfea893f6d78dad438276d2170654323a4256abf146639fc9\": container with ID starting with d8b6dfb90e15adbcfea893f6d78dad438276d2170654323a4256abf146639fc9 not found: ID does not exist" Oct 30 06:32:54.865039 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:32:54.865007 6999 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7312-1a493a\" is forbidden: User \"system:node:ip-10-0-131-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-226.ec2.internal' and this object" podUID="c4050c60-cfb6-456f-bc1c-42851a8fbdd5" pod="opendatahub/test-curl-7312-1a493a" Oct 30 06:32:55.091054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:55.090961 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:55.609855 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:55.609820 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4050c60-cfb6-456f-bc1c-42851a8fbdd5" path="/var/lib/kubelet/pods/c4050c60-cfb6-456f-bc1c-42851a8fbdd5/volumes" Oct 30 06:32:56.845779 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:56.845726 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:58.615999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:58.615972 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:32:58.791669 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:58.791641 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" Oct 30 06:32:58.795343 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:32:58.795318 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" Oct 30 06:33:00.464251 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:00.464166 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:00.812530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:00.812452 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" Oct 30 06:33:00.812683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:00.812539 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" Oct 30 06:33:02.190833 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:02.190797 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:02.334346 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:02.334316 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:04.060048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:04.059945 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:04.164060 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:33:04.164024 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:33:04.512564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:04.512532 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:05.848978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:05.848945 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:07.636495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:07.636461 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:09.345409 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:09.345380 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:10.988821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:10.988784 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg"] Oct 30 06:33:10.989605 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:10.989582 6999 cpu_manager.go:414] "containerMap: removing container" podUID="c4050c60-cfb6-456f-bc1c-42851a8fbdd5" containerName="test-curl-7312-1a493a" Oct 30 06:33:10.989605 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:10.989604 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4050c60-cfb6-456f-bc1c-42851a8fbdd5" containerName="test-curl-7312-1a493a" Oct 30 06:33:10.989771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:10.989623 6999 cpu_manager.go:414] "containerMap: removing container" podUID="97ead49a-073d-4bf3-bd25-afef28d6c83e" containerName="main" Oct 30 06:33:10.989771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:10.989629 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97ead49a-073d-4bf3-bd25-afef28d6c83e" containerName="main" Oct 30 06:33:10.989771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:10.989724 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4050c60-cfb6-456f-bc1c-42851a8fbdd5" containerName="test-curl-7312-1a493a" Oct 30 06:33:10.989771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:10.989734 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="97ead49a-073d-4bf3-bd25-afef28d6c83e" containerName="main" Oct 30 06:33:10.995020 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:10.995002 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:11.015040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.015016 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg"] Oct 30 06:33:11.024045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.024017 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-f0145f-serving-cert\"" Oct 30 06:33:11.045489 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.045461 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e88bddc-e0c1-4516-bd47-356b38380639-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:11.045642 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.045496 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfqmk\" (UniqueName: \"kubernetes.io/projected/1e88bddc-e0c1-4516-bd47-356b38380639-kube-api-access-qfqmk\") pod \"maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:11.045642 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.045547 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:11.045778 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.045700 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-metrics-certs\") pod \"maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:11.108663 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.108628 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:11.146708 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.146678 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e88bddc-e0c1-4516-bd47-356b38380639-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:11.146708 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.146713 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfqmk\" (UniqueName: \"kubernetes.io/projected/1e88bddc-e0c1-4516-bd47-356b38380639-kube-api-access-qfqmk\") pod \"maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:11.146892 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.146778 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:11.146962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.146945 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-metrics-certs\") pod \"maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:11.146999 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:33:11.146976 6999 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-serving-cert: secret "maas-api-e2e-extmod-a-f0145f-serving-cert" not found Oct 30 06:33:11.147072 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:33:11.147059 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-maas-api-tls podName:1e88bddc-e0c1-4516-bd47-356b38380639 nodeName:}" failed. No retries permitted until 2026-10-30 06:33:11.647037576 +0000 UTC m=+2358.156057460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-maas-api-tls") pod "maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" (UID: "1e88bddc-e0c1-4516-bd47-356b38380639") : secret "maas-api-e2e-extmod-a-f0145f-serving-cert" not found Oct 30 06:33:11.147423 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.147405 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e88bddc-e0c1-4516-bd47-356b38380639-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:11.149528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.149508 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-metrics-certs\") pod \"maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:11.170086 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.170068 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfqmk\" (UniqueName: \"kubernetes.io/projected/1e88bddc-e0c1-4516-bd47-356b38380639-kube-api-access-qfqmk\") pod \"maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:11.250561 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.250478 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:11.396937 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.396904 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:11.652840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.652812 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:11.655297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.655279 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-maas-api-tls\") pod \"maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:11.908768 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:11.908674 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:12.238587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:12.238550 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg"] Oct 30 06:33:12.239919 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:33:12.239888 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e88bddc_e0c1_4516_bd47_356b38380639.slice/crio-5d035b847301240ba435734c94d860d5c55b5cb6e8164db56d92bc88506390f3 WatchSource:0}: Error finding container 5d035b847301240ba435734c94d860d5c55b5cb6e8164db56d92bc88506390f3: Status 404 returned error can't find the container with id 5d035b847301240ba435734c94d860d5c55b5cb6e8164db56d92bc88506390f3 Oct 30 06:33:12.241883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:12.241859 6999 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:33:12.962709 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:12.962597 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" event={"ID":"1e88bddc-e0c1-4516-bd47-356b38380639","Type":"ContainerStarted","Data":"a8f5709f69eef6bb720308de728d176efe2f2380500eca8c121bd04a43c349ef"} Oct 30 06:33:12.962709 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:12.962663 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" event={"ID":"1e88bddc-e0c1-4516-bd47-356b38380639","Type":"ContainerStarted","Data":"5d035b847301240ba435734c94d860d5c55b5cb6e8164db56d92bc88506390f3"} Oct 30 06:33:12.962957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:12.962806 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:13.006546 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:13.006490 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" podStartSLOduration=2.585445268 podStartE2EDuration="3.006474782s" totalImagesPullingTime="421.029514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:13.005373046 +0000 UTC m=+2359.514392944" watchObservedRunningTime="2026-10-30 06:33:13.006474782 +0000 UTC m=+2359.515494670" Oct 30 06:33:13.049258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:13.049203 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:13.969703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:13.969670 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:14.942842 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:14.942778 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:16.748358 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.748326 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb"] Oct 30 06:33:16.752704 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.752681 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:16.767300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.767274 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-ad1661-serving-cert\"" Oct 30 06:33:16.782251 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.782200 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb"] Oct 30 06:33:16.824248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.824194 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6794b5-28bc-4b10-a0d4-c1f529a09908-metrics-certs\") pod \"maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:16.824484 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.824350 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6794b5-28bc-4b10-a0d4-c1f529a09908-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:16.824484 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.824411 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mq8g\" (UniqueName: \"kubernetes.io/projected/2e6794b5-28bc-4b10-a0d4-c1f529a09908-kube-api-access-4mq8g\") pod \"maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:16.824630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.824559 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6794b5-28bc-4b10-a0d4-c1f529a09908-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:16.867052 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.867013 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:16.925772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.925739 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6794b5-28bc-4b10-a0d4-c1f529a09908-metrics-certs\") pod \"maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:16.925943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.925789 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6794b5-28bc-4b10-a0d4-c1f529a09908-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:16.925943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.925825 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mq8g\" (UniqueName: \"kubernetes.io/projected/2e6794b5-28bc-4b10-a0d4-c1f529a09908-kube-api-access-4mq8g\") pod \"maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:16.925943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.925869 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6794b5-28bc-4b10-a0d4-c1f529a09908-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:16.926610 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.926573 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6794b5-28bc-4b10-a0d4-c1f529a09908-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:16.928856 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.928833 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6794b5-28bc-4b10-a0d4-c1f529a09908-metrics-certs\") pod \"maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:16.928998 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.928972 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6794b5-28bc-4b10-a0d4-c1f529a09908-maas-api-tls\") pod \"maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:16.949116 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:16.949091 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mq8g\" (UniqueName: \"kubernetes.io/projected/2e6794b5-28bc-4b10-a0d4-c1f529a09908-kube-api-access-4mq8g\") pod \"maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:17.012867 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:17.012719 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:17.066987 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:17.066955 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:17.158637 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:17.158598 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:17.381750 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:17.381724 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb"] Oct 30 06:33:17.383159 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:33:17.383134 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6794b5_28bc_4b10_a0d4_c1f529a09908.slice/crio-8773a98c033f3c970278176fd44da7e4b101ed7806c6ffdcf7bd93b239de40c1 WatchSource:0}: Error finding container 8773a98c033f3c970278176fd44da7e4b101ed7806c6ffdcf7bd93b239de40c1: Status 404 returned error can't find the container with id 8773a98c033f3c970278176fd44da7e4b101ed7806c6ffdcf7bd93b239de40c1 Oct 30 06:33:17.997641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:17.997548 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" event={"ID":"2e6794b5-28bc-4b10-a0d4-c1f529a09908","Type":"ContainerStarted","Data":"3b0cb8be02e8984fd68112bf0d371d59cb6be300a43aa189687d9eb1f2f86120"} Oct 30 06:33:17.997641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:17.997607 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" event={"ID":"2e6794b5-28bc-4b10-a0d4-c1f529a09908","Type":"ContainerStarted","Data":"8773a98c033f3c970278176fd44da7e4b101ed7806c6ffdcf7bd93b239de40c1"} Oct 30 06:33:17.997641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:17.997636 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:18.041796 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.041732 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" podStartSLOduration=1.725135981 podStartE2EDuration="2.041711376s" totalImagesPullingTime="316.575395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:18.040021609 +0000 UTC m=+2364.549041496" watchObservedRunningTime="2026-10-30 06:33:18.041711376 +0000 UTC m=+2364.550731263" Oct 30 06:33:18.547119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.547083 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms"] Oct 30 06:33:18.551583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.551561 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:18.580040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.580010 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-a0083e-serving-cert\"" Oct 30 06:33:18.585935 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.585902 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms"] Oct 30 06:33:18.650895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.650861 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx7zq\" (UniqueName: \"kubernetes.io/projected/115bfbd1-fef2-4d48-a261-a204e607f077-kube-api-access-mx7zq\") pod \"maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:18.650895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.650903 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/115bfbd1-fef2-4d48-a261-a204e607f077-metrics-certs\") pod \"maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:18.651101 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.651010 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/115bfbd1-fef2-4d48-a261-a204e607f077-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:18.651101 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.651062 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/115bfbd1-fef2-4d48-a261-a204e607f077-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:18.688787 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.688746 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:18.753156 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.753116 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx7zq\" (UniqueName: \"kubernetes.io/projected/115bfbd1-fef2-4d48-a261-a204e607f077-kube-api-access-mx7zq\") pod \"maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:18.753156 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.753168 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/115bfbd1-fef2-4d48-a261-a204e607f077-metrics-certs\") pod \"maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:18.753771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.753479 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/115bfbd1-fef2-4d48-a261-a204e607f077-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:18.753771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.753603 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/115bfbd1-fef2-4d48-a261-a204e607f077-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:18.755307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.754717 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/115bfbd1-fef2-4d48-a261-a204e607f077-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:18.760019 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.759990 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/115bfbd1-fef2-4d48-a261-a204e607f077-metrics-certs\") pod \"maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:18.760019 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.760012 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/115bfbd1-fef2-4d48-a261-a204e607f077-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:18.777747 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.777702 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx7zq\" (UniqueName: \"kubernetes.io/projected/115bfbd1-fef2-4d48-a261-a204e607f077-kube-api-access-mx7zq\") pod \"maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:18.833078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.832991 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:18.866401 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.866369 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:18.977151 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:18.977119 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:33:19.183990 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:19.183960 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms"] Oct 30 06:33:19.185327 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:33:19.185297 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod115bfbd1_fef2_4d48_a261_a204e607f077.slice/crio-649edbae27e9848b7c810f448075514092275eb0f794c10363c6b51bd7e4d997 WatchSource:0}: Error finding container 649edbae27e9848b7c810f448075514092275eb0f794c10363c6b51bd7e4d997: Status 404 returned error can't find the container with id 649edbae27e9848b7c810f448075514092275eb0f794c10363c6b51bd7e4d997 Oct 30 06:33:20.013851 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.013813 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" event={"ID":"115bfbd1-fef2-4d48-a261-a204e607f077","Type":"ContainerStarted","Data":"a4aeda6b027952647bfd9c1993b0cfb41f81588a26d959ee2e1f8c3c693ad0b7"} Oct 30 06:33:20.014158 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.013869 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" event={"ID":"115bfbd1-fef2-4d48-a261-a204e607f077","Type":"ContainerStarted","Data":"649edbae27e9848b7c810f448075514092275eb0f794c10363c6b51bd7e4d997"} Oct 30 06:33:20.014158 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.013975 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:20.056641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.056594 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" podStartSLOduration=1.75796689 podStartE2EDuration="2.056580568s" totalImagesPullingTime="298.613678ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:20.055569911 +0000 UTC m=+2366.564589821" watchObservedRunningTime="2026-10-30 06:33:20.056580568 +0000 UTC m=+2366.565600454" Oct 30 06:33:20.444568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.444537 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd"] Oct 30 06:33:20.447910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.447886 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:20.462475 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.462451 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-69f3fa-serving-cert\"" Oct 30 06:33:20.470840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.470810 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd"] Oct 30 06:33:20.571412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.571376 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:20.588478 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.588449 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fc83c12-1924-49f9-9725-901610b7ed41-maas-api-tls\") pod \"maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:20.588664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.588643 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc83c12-1924-49f9-9725-901610b7ed41-metrics-certs\") pod \"maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:20.588767 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.588749 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc83c12-1924-49f9-9725-901610b7ed41-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:20.588832 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.588820 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6mlt\" (UniqueName: \"kubernetes.io/projected/7fc83c12-1924-49f9-9725-901610b7ed41-kube-api-access-r6mlt\") pod \"maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:20.690361 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.690315 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc83c12-1924-49f9-9725-901610b7ed41-metrics-certs\") pod \"maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:20.690556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.690412 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc83c12-1924-49f9-9725-901610b7ed41-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:20.690556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.690546 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6mlt\" (UniqueName: \"kubernetes.io/projected/7fc83c12-1924-49f9-9725-901610b7ed41-kube-api-access-r6mlt\") pod \"maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:20.690728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.690709 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fc83c12-1924-49f9-9725-901610b7ed41-maas-api-tls\") pod \"maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:20.692414 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.692382 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc83c12-1924-49f9-9725-901610b7ed41-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:20.694726 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.694660 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc83c12-1924-49f9-9725-901610b7ed41-metrics-certs\") pod \"maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:20.694985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.694960 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fc83c12-1924-49f9-9725-901610b7ed41-maas-api-tls\") pod \"maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:20.714517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.714478 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6mlt\" (UniqueName: \"kubernetes.io/projected/7fc83c12-1924-49f9-9725-901610b7ed41-kube-api-access-r6mlt\") pod \"maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:20.720009 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.719969 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:20.761498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:20.761462 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:21.067957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:21.067927 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd"] Oct 30 06:33:21.069692 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:33:21.069667 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc83c12_1924_49f9_9725_901610b7ed41.slice/crio-acdc4b4184119e2add454d69b45aa223275ff8ade68c2a7d4e020bf78f30218f WatchSource:0}: Error finding container acdc4b4184119e2add454d69b45aa223275ff8ade68c2a7d4e020bf78f30218f: Status 404 returned error can't find the container with id acdc4b4184119e2add454d69b45aa223275ff8ade68c2a7d4e020bf78f30218f Oct 30 06:33:22.029453 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:22.029412 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" event={"ID":"7fc83c12-1924-49f9-9725-901610b7ed41","Type":"ContainerStarted","Data":"1db50eb82fe58468f79f9ab17e291fd128a47648d8f2a8ce52d5f9e56f5b20ad"} Oct 30 06:33:22.029878 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:22.029481 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" event={"ID":"7fc83c12-1924-49f9-9725-901610b7ed41","Type":"ContainerStarted","Data":"acdc4b4184119e2add454d69b45aa223275ff8ade68c2a7d4e020bf78f30218f"} Oct 30 06:33:22.029878 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:22.029510 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:22.077516 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:22.077429 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" podStartSLOduration=1.7223143319999998 podStartE2EDuration="2.077411849s" totalImagesPullingTime="355.097517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:22.074582809 +0000 UTC m=+2368.583602697" watchObservedRunningTime="2026-10-30 06:33:22.077411849 +0000 UTC m=+2368.586431736" Oct 30 06:33:22.468055 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:22.468018 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:24.012136 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:24.012108 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:33:24.347968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:24.347872 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:26.025995 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:26.025966 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:33:26.129100 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:26.129069 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:27.957528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:27.957494 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:28.041308 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:28.041279 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:33:28.551917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:28.551888 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:29.476153 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:29.476119 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:29.651578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:29.651533 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:30.081013 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:30.080977 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:30.238152 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:30.238116 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:31.546898 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:31.546857 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:32.244128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:32.244096 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:33.453066 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:33.453029 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:34.144518 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:33:34.144478 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:33:34.566911 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:34.566818 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:34.837150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:34.837042 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:35.577750 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:35.577716 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:36.288178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:36.288135 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:36.456025 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:36.455975 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:37.758643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:37.758608 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:39.644722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:39.644692 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:41.624672 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:41.624634 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:43.602067 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:43.602024 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:45.551466 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:45.551431 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:47.218878 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:47.218833 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:49.051683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:49.051647 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:51.000474 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:51.000445 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:52.293386 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:52.293352 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:52.764929 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:52.764891 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:53.768834 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:53.768805 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:54.238080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:54.238043 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:54.547047 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:54.546975 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:54.717126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:54.717077 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:55.137078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.137030 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr"] Oct 30 06:33:55.142970 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.142949 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.160106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.160082 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a0083e-openshift-default-dockercfg-zxj88\"" Oct 30 06:33:55.176806 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.176778 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr"] Oct 30 06:33:55.267245 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.267179 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.267439 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.267326 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be652fcd-c5e8-4086-a10a-f80ded1187f4-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.267489 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.267440 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.267489 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.267485 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97289\" (UniqueName: \"kubernetes.io/projected/be652fcd-c5e8-4086-a10a-f80ded1187f4-kube-api-access-97289\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.267610 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.267508 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.267674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.267661 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.267713 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.267682 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.267713 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.267700 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.267823 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.267718 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.368964 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.368931 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.369159 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.368996 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97289\" (UniqueName: \"kubernetes.io/projected/be652fcd-c5e8-4086-a10a-f80ded1187f4-kube-api-access-97289\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.369159 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.369033 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.369159 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.369125 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.369159 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.369154 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.369443 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.369182 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.369443 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.369208 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.369443 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.369313 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.369443 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.369389 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be652fcd-c5e8-4086-a10a-f80ded1187f4-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.369736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.369606 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.369736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.369706 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.369973 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.369949 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.370096 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.370070 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be652fcd-c5e8-4086-a10a-f80ded1187f4-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.370264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.370191 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.372297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.372277 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.372602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.372580 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.392975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.392915 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.392975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.392953 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97289\" (UniqueName: \"kubernetes.io/projected/be652fcd-c5e8-4086-a10a-f80ded1187f4-kube-api-access-97289\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.459086 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.459052 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:55.616672 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.616631 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g"] Oct 30 06:33:55.621756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.621731 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.641548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.641520 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g"] Oct 30 06:33:55.674302 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.674194 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.674504 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.674484 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85d5059-e45f-418d-ae2c-2712a3241086-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.674586 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.674519 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v22r4\" (UniqueName: \"kubernetes.io/projected/f85d5059-e45f-418d-ae2c-2712a3241086-kube-api-access-v22r4\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.674586 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.674564 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.674667 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.674650 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85d5059-e45f-418d-ae2c-2712a3241086-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.674915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.674892 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85d5059-e45f-418d-ae2c-2712a3241086-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.675040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.674939 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.675040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.674978 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.675040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.675025 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.718617 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.718586 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-ad1661-openshift-default-dockercfg-2bbrc\"" Oct 30 06:33:55.776145 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.776113 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85d5059-e45f-418d-ae2c-2712a3241086-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.776145 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.776154 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.776429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.776176 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.776429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.776201 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.776429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.776304 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.776429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.776336 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85d5059-e45f-418d-ae2c-2712a3241086-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.776429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.776361 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v22r4\" (UniqueName: \"kubernetes.io/projected/f85d5059-e45f-418d-ae2c-2712a3241086-kube-api-access-v22r4\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.776429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.776389 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.776429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.776419 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85d5059-e45f-418d-ae2c-2712a3241086-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.776790 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.776612 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.776790 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.776627 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.776790 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.776682 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.777058 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.776999 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.777238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.777199 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85d5059-e45f-418d-ae2c-2712a3241086-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.779468 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.779427 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85d5059-e45f-418d-ae2c-2712a3241086-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.779824 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.779800 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.781787 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.781767 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr"] Oct 30 06:33:55.784031 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:33:55.783836 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe652fcd_c5e8_4086_a10a_f80ded1187f4.slice/crio-5e9b8aaabb208db547efad0c85d6c66bdaed0e85604d1bd5774dba133104151b WatchSource:0}: Error finding container 5e9b8aaabb208db547efad0c85d6c66bdaed0e85604d1bd5774dba133104151b: Status 404 returned error can't find the container with id 5e9b8aaabb208db547efad0c85d6c66bdaed0e85604d1bd5774dba133104151b Oct 30 06:33:55.786941 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.786901 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:55.787337 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.787103 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:55.787337 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.787199 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:55.803925 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.803902 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85d5059-e45f-418d-ae2c-2712a3241086-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.804756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.804729 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v22r4\" (UniqueName: \"kubernetes.io/projected/f85d5059-e45f-418d-ae2c-2712a3241086-kube-api-access-v22r4\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:55.942560 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:55.942472 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:56.257966 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:56.257868 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" event={"ID":"be652fcd-c5e8-4086-a10a-f80ded1187f4","Type":"ContainerStarted","Data":"50e45b5fcb9fcdb880384d14609ab3560e3f623a88f16aeb7f7dcb47b6deb1a1"} Oct 30 06:33:56.257966 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:56.257945 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" event={"ID":"be652fcd-c5e8-4086-a10a-f80ded1187f4","Type":"ContainerStarted","Data":"5e9b8aaabb208db547efad0c85d6c66bdaed0e85604d1bd5774dba133104151b"} Oct 30 06:33:56.300306 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:56.300257 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g"] Oct 30 06:33:56.318992 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:56.310619 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" podStartSLOduration=1.310598315 podStartE2EDuration="1.310598315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:56.307958454 +0000 UTC m=+2402.816978353" watchObservedRunningTime="2026-10-30 06:33:56.310598315 +0000 UTC m=+2402.819618203" Oct 30 06:33:56.333261 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:56.333184 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:56.333426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:56.333315 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:56.333426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:56.333367 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:33:56.459839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:56.459801 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:33:56.461441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:56.461413 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:33:56.461592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:56.461464 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" podUID="be652fcd-c5e8-4086-a10a-f80ded1187f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 06:33:56.536366 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:56.536263 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:57.292339 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:57.292238 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" event={"ID":"f85d5059-e45f-418d-ae2c-2712a3241086","Type":"ContainerStarted","Data":"6d5a063896c180814f634680a60853b68224317f34acb1e455a4c881b16b0fa3"} Oct 30 06:33:57.292339 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:57.292297 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" event={"ID":"f85d5059-e45f-418d-ae2c-2712a3241086","Type":"ContainerStarted","Data":"4eef34ab355fcc5625c72e22ed82b089cbc3cc8c7ff8598d22d3b9b46f39c109"} Oct 30 06:33:57.464240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:57.463400 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:33:57.464240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:57.463463 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" podUID="be652fcd-c5e8-4086-a10a-f80ded1187f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 06:33:57.993370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:57.991120 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:33:57.993370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:57.992480 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:33:57.993370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:57.992548 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" podUID="f85d5059-e45f-418d-ae2c-2712a3241086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 06:33:58.404434 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:58.403763 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:58.460102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:58.459630 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:33:58.460102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:58.459696 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" podUID="be652fcd-c5e8-4086-a10a-f80ded1187f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 06:33:58.551647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:58.551613 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:58.946238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:58.943718 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:33:58.946238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:58.943783 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" podUID="f85d5059-e45f-418d-ae2c-2712a3241086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 06:33:58.964805 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:58.964772 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:33:59.468240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:59.467655 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:33:59.468240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:59.467734 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" podUID="be652fcd-c5e8-4086-a10a-f80ded1187f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 06:33:59.953240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:59.946629 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:33:59.953240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:33:59.946689 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" podUID="f85d5059-e45f-418d-ae2c-2712a3241086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 06:34:00.475245 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:00.459660 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:34:00.475245 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:00.459725 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" podUID="be652fcd-c5e8-4086-a10a-f80ded1187f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 06:34:00.811266 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:00.811120 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:00.948257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:00.947689 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:34:00.948257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:00.947756 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" podUID="f85d5059-e45f-418d-ae2c-2712a3241086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 06:34:01.097294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:01.096780 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:01.403115 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:01.403077 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:01.473408 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:01.473363 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:34:01.473648 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:01.473433 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" podUID="be652fcd-c5e8-4086-a10a-f80ded1187f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 06:34:01.952688 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:01.952648 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body= Oct 30 06:34:01.953352 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:01.953320 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" podUID="f85d5059-e45f-418d-ae2c-2712a3241086" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" Oct 30 06:34:01.986504 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:01.986460 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:02.144230 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:02.144181 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:02.612016 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:02.611983 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:02.695330 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:02.695296 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:34:02.696324 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:02.696295 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:34:02.698782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:02.698713 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:34:02.768149 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:02.768079 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" podStartSLOduration=7.768058867 podStartE2EDuration="7.768058867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:57.363496842 +0000 UTC m=+2403.872516745" watchObservedRunningTime="2026-10-30 06:34:02.768058867 +0000 UTC m=+2409.277078755" Oct 30 06:34:02.952479 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:02.952391 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:34:02.952792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:02.952767 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:34:02.953528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:02.953508 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:34:03.430091 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:03.430056 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:03.604752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:03.604712 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:03.774373 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:03.774281 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:03.944623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:03.944584 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:04.111142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:04.111107 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:04.148174 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:34:04.148081 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:34:04.297756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:04.297720 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:04.444685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:04.444592 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:04.944797 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:04.944765 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:05.172940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:05.172911 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:05.328256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:05.328142 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:05.533331 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:05.533291 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:05.971593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:05.971548 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:06.117960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:06.117924 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:07.457154 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:07.457120 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:07.635668 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:07.635637 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:08.411422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:08.411388 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:08.625158 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:08.625126 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:08.801414 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:08.801319 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:09.483094 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:09.483059 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:11.038810 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:11.038776 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:11.195898 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:11.195863 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:11.353880 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:11.353774 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:12.902079 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:12.902038 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6"] Oct 30 06:34:12.906776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:12.906755 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:12.937040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:12.937000 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-a88d7d-serving-cert\"" Oct 30 06:34:12.941080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:12.941040 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6"] Oct 30 06:34:13.034282 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.034205 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:13.039969 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.039941 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-metrics-certs\") pod \"maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:13.040111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.040082 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj6gl\" (UniqueName: \"kubernetes.io/projected/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-kube-api-access-cj6gl\") pod \"maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:13.040226 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.040193 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-openshift-service-ca\") pod \"maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:13.040300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.040284 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-maas-api-tls\") pod \"maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:13.141723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.141688 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-metrics-certs\") pod \"maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:13.141928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.141820 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj6gl\" (UniqueName: \"kubernetes.io/projected/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-kube-api-access-cj6gl\") pod \"maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:13.141928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.141896 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-openshift-service-ca\") pod \"maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:13.142052 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.141953 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-maas-api-tls\") pod \"maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:13.142149 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:34:13.142126 6999 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-serving-cert: secret "maas-api-e2e-autores-a88d7d-serving-cert" not found Oct 30 06:34:13.142249 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:34:13.142235 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-maas-api-tls podName:1bca0367-1f0a-4d33-8bcc-551a7c5d1bda nodeName:}" failed. No retries permitted until 2026-10-30 06:34:13.642194792 +0000 UTC m=+2420.151214664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-maas-api-tls") pod "maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" (UID: "1bca0367-1f0a-4d33-8bcc-551a7c5d1bda") : secret "maas-api-e2e-autores-a88d7d-serving-cert" not found Oct 30 06:34:13.142635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.142613 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-openshift-service-ca\") pod \"maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:13.144584 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.144559 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-metrics-certs\") pod \"maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:13.168046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.167978 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj6gl\" (UniqueName: \"kubernetes.io/projected/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-kube-api-access-cj6gl\") pod \"maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:13.170084 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.170052 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:13.321151 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.321108 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:13.649398 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.649363 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-maas-api-tls\") pod \"maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:13.652292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.652262 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-maas-api-tls\") pod \"maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:13.822000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:13.821952 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:14.153664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:14.153631 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6"] Oct 30 06:34:14.154938 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:34:14.154906 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bca0367_1f0a_4d33_8bcc_551a7c5d1bda.slice/crio-9a34d68831cb5d47964cea642e48c5eef36e16ee8fe86d5fbf3534282cae81ba WatchSource:0}: Error finding container 9a34d68831cb5d47964cea642e48c5eef36e16ee8fe86d5fbf3534282cae81ba: Status 404 returned error can't find the container with id 9a34d68831cb5d47964cea642e48c5eef36e16ee8fe86d5fbf3534282cae81ba Oct 30 06:34:14.413463 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:14.413377 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" event={"ID":"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda","Type":"ContainerStarted","Data":"9a34d68831cb5d47964cea642e48c5eef36e16ee8fe86d5fbf3534282cae81ba"} Oct 30 06:34:15.067120 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:15.067013 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:15.422346 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:15.422309 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" event={"ID":"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda","Type":"ContainerStarted","Data":"840032b60be0389171423b434774858499b3e0aa1450d617ba2a0974abd18868"} Oct 30 06:34:15.422795 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:15.422415 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:15.466492 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:15.466438 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" podStartSLOduration=3.026730113 podStartE2EDuration="3.46642192s" totalImagesPullingTime="439.691807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:15.464191821 +0000 UTC m=+2421.973211708" watchObservedRunningTime="2026-10-30 06:34:15.46642192 +0000 UTC m=+2421.975441808" Oct 30 06:34:17.038474 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:17.038434 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:18.782948 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:18.782918 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:20.522076 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:20.522042 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:21.436551 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:21.436525 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:34:22.342822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:22.342790 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:24.110722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:24.110679 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:25.254354 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.254315 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj"] Oct 30 06:34:25.259156 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.259138 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.283166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.283134 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj"] Oct 30 06:34:25.334562 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.334523 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.334733 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.334651 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.334803 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.334785 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.334923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.334906 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.334975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.334958 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.335109 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.335088 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7dj7\" (UniqueName: \"kubernetes.io/projected/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-kube-api-access-d7dj7\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.335152 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.335142 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.335281 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.335261 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.335336 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.335322 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.437149 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.437111 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.437349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.437227 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.437349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.437265 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.437349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.437333 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dj7\" (UniqueName: \"kubernetes.io/projected/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-kube-api-access-d7dj7\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.437530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.437369 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.437530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.437429 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.437530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.437474 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.437686 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.437585 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.437686 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.437602 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.437771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.437673 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.437771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.437719 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.437771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.437757 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.437875 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.437775 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.439256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.439188 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.440426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.440397 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.440599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.440581 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.461174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.461144 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.461360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.461267 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dj7\" (UniqueName: \"kubernetes.io/projected/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-kube-api-access-d7dj7\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.574667 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.574568 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:25.902827 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.902803 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj"] Oct 30 06:34:25.904319 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:34:25.904292 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc0b89a_0fb5_4632_b6e2_d8586cd93435.slice/crio-60db96c3b755fdfdfaf457e26bdb5badf3770d13b1df45ff8d44ead6d802e94f WatchSource:0}: Error finding container 60db96c3b755fdfdfaf457e26bdb5badf3770d13b1df45ff8d44ead6d802e94f: Status 404 returned error can't find the container with id 60db96c3b755fdfdfaf457e26bdb5badf3770d13b1df45ff8d44ead6d802e94f Oct 30 06:34:25.906656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.906561 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:25.907558 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.907525 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:25.907659 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.907607 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:25.907720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:25.907654 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:26.495413 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:26.495372 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" event={"ID":"0bc0b89a-0fb5-4632-b6e2-d8586cd93435","Type":"ContainerStarted","Data":"c520d227fef6f43bb0c39ff7df673d827acfab5fe3ad7bb225fac714cf2812cf"} Oct 30 06:34:26.495825 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:26.495433 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" event={"ID":"0bc0b89a-0fb5-4632-b6e2-d8586cd93435","Type":"ContainerStarted","Data":"60db96c3b755fdfdfaf457e26bdb5badf3770d13b1df45ff8d44ead6d802e94f"} Oct 30 06:34:26.565067 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:26.565004 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" podStartSLOduration=1.564985206 podStartE2EDuration="1.564985206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:26.560693101 +0000 UTC m=+2433.069713012" watchObservedRunningTime="2026-10-30 06:34:26.564985206 +0000 UTC m=+2433.074005094" Oct 30 06:34:26.575051 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:26.575025 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:26.576816 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:26.576794 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:34:26.576905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:26.576840 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" podUID="0bc0b89a-0fb5-4632-b6e2-d8586cd93435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 06:34:27.575877 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:27.575837 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:34:27.576311 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:27.575893 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" podUID="0bc0b89a-0fb5-4632-b6e2-d8586cd93435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 06:34:27.766786 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:27.766702 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:28.575737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:28.575700 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:34:28.575922 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:28.575754 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" podUID="0bc0b89a-0fb5-4632-b6e2-d8586cd93435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 06:34:29.575811 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:29.575773 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:34:29.576005 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:29.575823 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" podUID="0bc0b89a-0fb5-4632-b6e2-d8586cd93435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 06:34:29.664700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:29.664672 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:30.575281 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:30.575244 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:34:30.575475 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:30.575302 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" podUID="0bc0b89a-0fb5-4632-b6e2-d8586cd93435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 06:34:31.465120 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:31.465079 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:31.575582 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:31.575550 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:34:31.575769 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:31.575614 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" podUID="0bc0b89a-0fb5-4632-b6e2-d8586cd93435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 06:34:32.580089 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:32.580058 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:32.580560 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:32.580450 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:32.581181 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:32.581160 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:34:33.250395 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:33.250363 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:34.144526 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:34:34.144489 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:34:35.051497 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:35.051455 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:36.868075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:36.868043 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:38.426462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.426417 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj"] Oct 30 06:34:38.431931 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.431911 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:38.451386 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.451356 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-2d8df2-serving-cert\"" Oct 30 06:34:38.463804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.463769 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj"] Oct 30 06:34:38.525938 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.525907 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b7d49d-e508-4a13-bc59-06a17f7e17fe-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:38.526150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.526124 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b7d49d-e508-4a13-bc59-06a17f7e17fe-metrics-certs\") pod \"maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:38.526588 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.526224 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b7d49d-e508-4a13-bc59-06a17f7e17fe-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:38.526588 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.526260 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97nzp\" (UniqueName: \"kubernetes.io/projected/70b7d49d-e508-4a13-bc59-06a17f7e17fe-kube-api-access-97nzp\") pod \"maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:38.558999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.558966 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:38.628116 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.628084 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b7d49d-e508-4a13-bc59-06a17f7e17fe-metrics-certs\") pod \"maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:38.628332 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.628131 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b7d49d-e508-4a13-bc59-06a17f7e17fe-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:38.628332 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.628148 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97nzp\" (UniqueName: \"kubernetes.io/projected/70b7d49d-e508-4a13-bc59-06a17f7e17fe-kube-api-access-97nzp\") pod \"maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:38.628332 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.628288 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b7d49d-e508-4a13-bc59-06a17f7e17fe-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:38.628830 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.628802 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b7d49d-e508-4a13-bc59-06a17f7e17fe-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:38.630700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.630678 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b7d49d-e508-4a13-bc59-06a17f7e17fe-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:38.630786 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.630772 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b7d49d-e508-4a13-bc59-06a17f7e17fe-metrics-certs\") pod \"maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:38.656080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.654595 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97nzp\" (UniqueName: \"kubernetes.io/projected/70b7d49d-e508-4a13-bc59-06a17f7e17fe-kube-api-access-97nzp\") pod \"maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:38.703967 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.703891 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:38.747932 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:38.747908 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:39.057080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:39.057057 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj"] Oct 30 06:34:39.059546 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:34:39.059523 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b7d49d_e508_4a13_bc59_06a17f7e17fe.slice/crio-b16f14252ae184b9c625ae43b360fbe98c8fe35b1ee1beba6315bfa8e51dd4bd WatchSource:0}: Error finding container b16f14252ae184b9c625ae43b360fbe98c8fe35b1ee1beba6315bfa8e51dd4bd: Status 404 returned error can't find the container with id b16f14252ae184b9c625ae43b360fbe98c8fe35b1ee1beba6315bfa8e51dd4bd Oct 30 06:34:39.584978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:39.584936 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" event={"ID":"70b7d49d-e508-4a13-bc59-06a17f7e17fe","Type":"ContainerStarted","Data":"fa79f8eb68f41b6e0041b1ce6c86758243be0daa91d076543f788a0c25750aad"} Oct 30 06:34:39.585487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:39.585003 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" event={"ID":"70b7d49d-e508-4a13-bc59-06a17f7e17fe","Type":"ContainerStarted","Data":"b16f14252ae184b9c625ae43b360fbe98c8fe35b1ee1beba6315bfa8e51dd4bd"} Oct 30 06:34:39.585487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:39.585053 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:39.628360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:39.628311 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" podStartSLOduration=1.223362162 podStartE2EDuration="1.628297016s" totalImagesPullingTime="404.934854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:39.62654933 +0000 UTC m=+2446.135569217" watchObservedRunningTime="2026-10-30 06:34:39.628297016 +0000 UTC m=+2446.137316902" Oct 30 06:34:40.326582 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.326551 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4"] Oct 30 06:34:40.331194 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.331172 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.361577 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.361551 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4"] Oct 30 06:34:40.454925 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.454893 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.455105 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.454937 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.455105 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.455069 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.455184 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.455109 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n79gb\" (UniqueName: \"kubernetes.io/projected/567e2929-da18-45e4-a1e4-4e86f1ad1855-kube-api-access-n79gb\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.455184 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.455138 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.455297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.455243 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567e2929-da18-45e4-a1e4-4e86f1ad1855-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.455297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.455288 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.455434 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.455414 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.455554 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.455475 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.557308 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.557277 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.557517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.557353 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.557517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.557381 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n79gb\" (UniqueName: \"kubernetes.io/projected/567e2929-da18-45e4-a1e4-4e86f1ad1855-kube-api-access-n79gb\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.557517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.557401 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.557517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.557451 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567e2929-da18-45e4-a1e4-4e86f1ad1855-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.557517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.557511 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.557793 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.557611 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.557793 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.557659 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.557793 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.557707 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.557968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.557892 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-data\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.558046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.558023 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-workload-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.558106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.558087 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-credential-socket\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.558168 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.558119 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-workload-certs\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.558372 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.558316 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567e2929-da18-45e4-a1e4-4e86f1ad1855-istiod-ca-cert\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.559972 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.559950 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-envoy\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.560192 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.560176 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-podinfo\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.562968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.562935 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:40.580669 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.580597 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-token\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.580839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.580812 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n79gb\" (UniqueName: \"kubernetes.io/projected/567e2929-da18-45e4-a1e4-4e86f1ad1855-kube-api-access-n79gb\") pod \"e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.646043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.646013 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:40.756129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.756099 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6"] Oct 30 06:34:40.762059 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.762032 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.785134 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.785089 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6"] Oct 30 06:34:40.861963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.861924 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.862145 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.861968 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd69357f-342b-4367-96c0-a98fa610742d-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.862145 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.862001 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd69357f-342b-4367-96c0-a98fa610742d-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.862325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.862208 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.862325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.862299 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.862426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.862332 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.862426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.862357 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsr2h\" (UniqueName: \"kubernetes.io/projected/dd69357f-342b-4367-96c0-a98fa610742d-kube-api-access-wsr2h\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.862426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.862374 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd69357f-342b-4367-96c0-a98fa610742d-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.862426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.862392 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.963696 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.963647 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.963902 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.963728 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.963902 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.963758 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.963902 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.963784 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsr2h\" (UniqueName: \"kubernetes.io/projected/dd69357f-342b-4367-96c0-a98fa610742d-kube-api-access-wsr2h\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.963902 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.963811 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd69357f-342b-4367-96c0-a98fa610742d-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.963902 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.963840 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.964599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.963916 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.964599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.963940 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd69357f-342b-4367-96c0-a98fa610742d-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.964599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.964069 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd69357f-342b-4367-96c0-a98fa610742d-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.964599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.964200 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.964599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.964550 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.964599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.964566 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.964892 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.964684 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.965066 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.965047 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd69357f-342b-4367-96c0-a98fa610742d-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.966441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.966415 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.966568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.966467 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd69357f-342b-4367-96c0-a98fa610742d-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.973775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.973745 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4"] Oct 30 06:34:40.975954 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:34:40.975922 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567e2929_da18_45e4_a1e4_4e86f1ad1855.slice/crio-89e6b330929926adcffae8621324671ba8d4e11803aa0f7778de187382c1c19f WatchSource:0}: Error finding container 89e6b330929926adcffae8621324671ba8d4e11803aa0f7778de187382c1c19f: Status 404 returned error can't find the container with id 89e6b330929926adcffae8621324671ba8d4e11803aa0f7778de187382c1c19f Oct 30 06:34:40.979148 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.979109 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:40.979293 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.979181 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:40.979293 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.979264 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:40.987762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.987731 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd69357f-342b-4367-96c0-a98fa610742d-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:40.987956 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:40.987933 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsr2h\" (UniqueName: \"kubernetes.io/projected/dd69357f-342b-4367-96c0-a98fa610742d-kube-api-access-wsr2h\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:41.082348 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:41.082319 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:41.414780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:41.414756 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6"] Oct 30 06:34:41.416753 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:34:41.416727 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd69357f_342b_4367_96c0_a98fa610742d.slice/crio-e323d23d9ce3355a83b0b2c7bbafc962e1afda04cf1280b8c8ad369bae029d39 WatchSource:0}: Error finding container e323d23d9ce3355a83b0b2c7bbafc962e1afda04cf1280b8c8ad369bae029d39: Status 404 returned error can't find the container with id e323d23d9ce3355a83b0b2c7bbafc962e1afda04cf1280b8c8ad369bae029d39 Oct 30 06:34:41.420262 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:41.420182 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:41.420343 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:41.420315 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:41.420389 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:41.420360 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:41.611407 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:41.611366 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" event={"ID":"dd69357f-342b-4367-96c0-a98fa610742d","Type":"ContainerStarted","Data":"a49b54f14ee5bf73e1e12e837ae811b86ef5e591681769b5b6cf018bff66f6ae"} Oct 30 06:34:41.611613 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:41.611441 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" event={"ID":"dd69357f-342b-4367-96c0-a98fa610742d","Type":"ContainerStarted","Data":"e323d23d9ce3355a83b0b2c7bbafc962e1afda04cf1280b8c8ad369bae029d39"} Oct 30 06:34:41.622839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:41.622808 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" event={"ID":"567e2929-da18-45e4-a1e4-4e86f1ad1855","Type":"ContainerStarted","Data":"8d8b4768453bc5d1391d988e6b836c8fa81e9d69885ed9f432da054567e0fa66"} Oct 30 06:34:41.623023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:41.622857 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" event={"ID":"567e2929-da18-45e4-a1e4-4e86f1ad1855","Type":"ContainerStarted","Data":"89e6b330929926adcffae8621324671ba8d4e11803aa0f7778de187382c1c19f"} Oct 30 06:34:41.646536 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:41.646506 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:41.657139 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:41.657079 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" podStartSLOduration=1.6570610829999999 podStartE2EDuration="1.657061083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:41.655821923 +0000 UTC m=+2448.164841825" watchObservedRunningTime="2026-10-30 06:34:41.657061083 +0000 UTC m=+2448.166080971" Oct 30 06:34:41.703704 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:41.703587 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" podStartSLOduration=1.70356692 podStartE2EDuration="1.70356692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:41.699912995 +0000 UTC m=+2448.208932887" watchObservedRunningTime="2026-10-30 06:34:41.70356692 +0000 UTC m=+2448.212586808" Oct 30 06:34:42.085726 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:42.085628 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:42.088082 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:42.088052 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:34:42.088345 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:42.088323 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" podUID="dd69357f-342b-4367-96c0-a98fa610742d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 06:34:42.655399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:42.655358 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:34:42.656020 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:42.655988 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" podUID="567e2929-da18-45e4-a1e4-4e86f1ad1855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 06:34:42.711684 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:42.711638 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:43.084838 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:43.084744 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:34:43.085086 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:43.085062 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" podUID="dd69357f-342b-4367-96c0-a98fa610742d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 06:34:43.655243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:43.652292 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:34:43.655243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:43.652344 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" podUID="567e2929-da18-45e4-a1e4-4e86f1ad1855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 06:34:44.084850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:44.084733 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:34:44.084850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:44.084806 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" podUID="dd69357f-342b-4367-96c0-a98fa610742d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 06:34:44.647051 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:44.647006 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:34:44.647384 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:44.647338 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" podUID="567e2929-da18-45e4-a1e4-4e86f1ad1855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 06:34:44.990300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:44.990042 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:45.086239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:45.085585 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:34:45.086239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:45.085669 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" podUID="dd69357f-342b-4367-96c0-a98fa610742d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 06:34:45.611107 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:45.611069 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:34:45.648593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:45.646834 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:34:45.648593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:45.646899 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" podUID="567e2929-da18-45e4-a1e4-4e86f1ad1855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 06:34:46.094303 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:46.089580 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:34:46.094303 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:46.089646 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" podUID="dd69357f-342b-4367-96c0-a98fa610742d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 06:34:46.648240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:46.647694 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body= Oct 30 06:34:46.648240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:46.647762 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" podUID="567e2929-da18-45e4-a1e4-4e86f1ad1855" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" Oct 30 06:34:47.089886 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:47.089629 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 06:34:47.089886 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:47.089731 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" podUID="dd69357f-342b-4367-96c0-a98fa610742d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 06:34:47.133104 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:47.133074 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:47.991048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:47.991014 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:47.991485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:47.991463 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:47.992932 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:47.992911 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:34:48.088997 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:48.088965 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:48.089283 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:48.089255 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:48.089996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:48.089975 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:34:48.875744 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:48.875705 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq"] Oct 30 06:34:48.880296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:48.880277 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:48.909869 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:48.909824 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-6d1778-serving-cert\"" Oct 30 06:34:48.916630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:48.916591 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq"] Oct 30 06:34:48.953262 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:48.953200 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c821205-038e-4dd0-b122-837935d53312-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:48.953446 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:48.953340 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c821205-038e-4dd0-b122-837935d53312-metrics-certs\") pod \"maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:48.953491 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:48.953463 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c821205-038e-4dd0-b122-837935d53312-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:48.953538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:48.953514 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wrb5\" (UniqueName: \"kubernetes.io/projected/4c821205-038e-4dd0-b122-837935d53312-kube-api-access-6wrb5\") pod \"maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:49.005276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:49.005238 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:49.055066 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:49.055023 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c821205-038e-4dd0-b122-837935d53312-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:49.055349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:49.055094 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wrb5\" (UniqueName: \"kubernetes.io/projected/4c821205-038e-4dd0-b122-837935d53312-kube-api-access-6wrb5\") pod \"maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:49.055349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:49.055320 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c821205-038e-4dd0-b122-837935d53312-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:49.055517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:49.055398 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c821205-038e-4dd0-b122-837935d53312-metrics-certs\") pod \"maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:49.055863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:49.055838 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c821205-038e-4dd0-b122-837935d53312-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:49.057980 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:49.057955 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c821205-038e-4dd0-b122-837935d53312-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:49.058064 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:49.058002 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c821205-038e-4dd0-b122-837935d53312-metrics-certs\") pod \"maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:49.078589 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:49.078555 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wrb5\" (UniqueName: \"kubernetes.io/projected/4c821205-038e-4dd0-b122-837935d53312-kube-api-access-6wrb5\") pod \"maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:49.169063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:49.168970 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:49.195731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:49.195696 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:49.528072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:49.528044 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq"] Oct 30 06:34:49.529402 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:34:49.529377 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c821205_038e_4dd0_b122_837935d53312.slice/crio-76570b166770f1f41b4a21143b73ec4d32925461734ba6959f2c8b160e27b24a WatchSource:0}: Error finding container 76570b166770f1f41b4a21143b73ec4d32925461734ba6959f2c8b160e27b24a: Status 404 returned error can't find the container with id 76570b166770f1f41b4a21143b73ec4d32925461734ba6959f2c8b160e27b24a Oct 30 06:34:49.739416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:49.739373 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" event={"ID":"4c821205-038e-4dd0-b122-837935d53312","Type":"ContainerStarted","Data":"76570b166770f1f41b4a21143b73ec4d32925461734ba6959f2c8b160e27b24a"} Oct 30 06:34:50.747784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:50.747743 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" event={"ID":"4c821205-038e-4dd0-b122-837935d53312","Type":"ContainerStarted","Data":"ac82548fe44476e02eee78ce29ab10e029b53ccb57e0b09ae6361787a0bb56e8"} Oct 30 06:34:50.748231 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:50.747836 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:50.791252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:50.791186 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" podStartSLOduration=2.478625622 podStartE2EDuration="2.791172306s" totalImagesPullingTime="312.546684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:50.788618182 +0000 UTC m=+2457.297638070" watchObservedRunningTime="2026-10-30 06:34:50.791172306 +0000 UTC m=+2457.300192230" Oct 30 06:34:51.010933 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:51.010836 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:51.874723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:51.874692 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:52.749528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:52.749461 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:54.554090 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:54.554054 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:55.808585 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:55.808543 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k"] Oct 30 06:34:55.814389 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:55.814363 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:55.845291 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:55.845262 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k"] Oct 30 06:34:55.989829 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:55.989792 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:55.990023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:55.989968 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:55.990101 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:55.990081 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86cc6726-5087-432a-8487-5dd10dfb4c51-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:55.990152 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:55.990117 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:55.990198 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:55.990164 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:55.990284 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:55.990267 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:55.990370 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:55.990356 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:55.990443 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:55.990416 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snpvf\" (UniqueName: \"kubernetes.io/projected/86cc6726-5087-432a-8487-5dd10dfb4c51-kube-api-access-snpvf\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:55.990531 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:55.990512 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.092428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.092338 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.092428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.092380 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.092428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.092429 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.092661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.092474 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snpvf\" (UniqueName: \"kubernetes.io/projected/86cc6726-5087-432a-8487-5dd10dfb4c51-kube-api-access-snpvf\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.092661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.092531 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.092661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.092627 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.092804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.092733 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.092863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.092811 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-workload-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.092863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.092827 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86cc6726-5087-432a-8487-5dd10dfb4c51-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.092863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.092835 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-data\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.093103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.092868 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.093103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.092950 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-credential-socket\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.093103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.093084 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-workload-certs\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.093576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.093505 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86cc6726-5087-432a-8487-5dd10dfb4c51-istiod-ca-cert\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.095122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.095095 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-envoy\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.095311 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.095295 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-podinfo\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.116415 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.116392 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-token\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.116561 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.116479 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snpvf\" (UniqueName: \"kubernetes.io/projected/86cc6726-5087-432a-8487-5dd10dfb4c51-kube-api-access-snpvf\") pod \"e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.129543 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.129514 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:56.341116 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.341077 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:56.459735 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.459698 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k"] Oct 30 06:34:56.460989 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:34:56.460952 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86cc6726_5087_432a_8487_5dd10dfb4c51.slice/crio-c44d90bd130b42ce558fc251e1a43a792df4bd0a4d13868d7adf612e9c2b5260 WatchSource:0}: Error finding container c44d90bd130b42ce558fc251e1a43a792df4bd0a4d13868d7adf612e9c2b5260: Status 404 returned error can't find the container with id c44d90bd130b42ce558fc251e1a43a792df4bd0a4d13868d7adf612e9c2b5260 Oct 30 06:34:56.463996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.463960 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:56.464104 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.464042 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:56.464104 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.464090 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:34:56.760244 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.760149 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:34:56.789256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.789191 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" event={"ID":"86cc6726-5087-432a-8487-5dd10dfb4c51","Type":"ContainerStarted","Data":"2baf2d096c4d9ca05d309281b63164c94225cb419e1aee7fc71b475ab1764192"} Oct 30 06:34:56.789467 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.789285 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" event={"ID":"86cc6726-5087-432a-8487-5dd10dfb4c51","Type":"ContainerStarted","Data":"c44d90bd130b42ce558fc251e1a43a792df4bd0a4d13868d7adf612e9c2b5260"} Oct 30 06:34:56.851344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:56.849551 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" podStartSLOduration=1.8495329 podStartE2EDuration="1.8495329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:56.846771853 +0000 UTC m=+2463.355791752" watchObservedRunningTime="2026-10-30 06:34:56.8495329 +0000 UTC m=+2463.358552781" Oct 30 06:34:57.130441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:57.130395 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:34:57.132460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:57.132435 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 06:34:57.132544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:57.132490 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" podUID="86cc6726-5087-432a-8487-5dd10dfb4c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 06:34:58.130075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:58.130037 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 06:34:58.130601 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:58.130094 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" podUID="86cc6726-5087-432a-8487-5dd10dfb4c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 06:34:58.257754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:58.257696 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:34:59.130639 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:59.130596 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 06:34:59.131064 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:34:59.130674 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" podUID="86cc6726-5087-432a-8487-5dd10dfb4c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 06:35:00.130639 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.130603 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 06:35:00.131164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.130665 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" podUID="86cc6726-5087-432a-8487-5dd10dfb4c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 06:35:00.254816 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.254781 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:00.386509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.386407 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9"] Oct 30 06:35:00.391563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.391539 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.427486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.427455 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9"] Oct 30 06:35:00.484496 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.484452 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-envoy\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.484660 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.484517 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-workload-certs\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.484713 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.484687 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-token\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.484834 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.484806 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-podinfo\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.484887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.484871 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6hbv\" (UniqueName: \"kubernetes.io/projected/7f98765a-50a6-46b3-9931-024c9bd7934c-kube-api-access-f6hbv\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.484926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.484914 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-credential-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.485037 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.485014 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f98765a-50a6-46b3-9931-024c9bd7934c-istiod-ca-cert\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.485093 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.485075 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-data\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.485296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.485263 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-workload-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.494362 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.494338 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-a88d7d-openshift-default-dockercfg-s6dkq\"" Oct 30 06:35:00.587026 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.586988 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-credential-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.587026 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.587034 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f98765a-50a6-46b3-9931-024c9bd7934c-istiod-ca-cert\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.587390 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.587067 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-data\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.587390 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.587146 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-workload-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.587390 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.587271 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-envoy\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.587390 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.587317 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-workload-certs\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.587632 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.587407 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-token\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.587632 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.587471 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-podinfo\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.587632 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.587511 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6hbv\" (UniqueName: \"kubernetes.io/projected/7f98765a-50a6-46b3-9931-024c9bd7934c-kube-api-access-f6hbv\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.587632 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.587567 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-credential-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.587870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.587780 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f98765a-50a6-46b3-9931-024c9bd7934c-istiod-ca-cert\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.587870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.587803 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-data\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.587870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.587839 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-workload-socket\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.588046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.587881 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-workload-certs\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.590178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.590153 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-envoy\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.590304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.590155 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-podinfo\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.611072 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.611039 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-token\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.611290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.611117 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6hbv\" (UniqueName: \"kubernetes.io/projected/7f98765a-50a6-46b3-9931-024c9bd7934c-kube-api-access-f6hbv\") pod \"e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:00.724121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:00.724031 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:01.079462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:01.079433 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9"] Oct 30 06:35:01.080627 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:35:01.080596 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f98765a_50a6_46b3_9931_024c9bd7934c.slice/crio-f2559b471ffaa91802d141a0c953004fe083e2f2d8aac7a75644e44da93a7478 WatchSource:0}: Error finding container f2559b471ffaa91802d141a0c953004fe083e2f2d8aac7a75644e44da93a7478: Status 404 returned error can't find the container with id f2559b471ffaa91802d141a0c953004fe083e2f2d8aac7a75644e44da93a7478 Oct 30 06:35:01.083641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:01.083606 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:01.083727 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:01.083683 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:01.083727 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:01.083714 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:01.151246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:01.150411 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 06:35:01.151246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:01.150471 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" podUID="86cc6726-5087-432a-8487-5dd10dfb4c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 06:35:01.861128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:01.861085 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" event={"ID":"7f98765a-50a6-46b3-9931-024c9bd7934c","Type":"ContainerStarted","Data":"3befa12c347c86e19ff801063898dc89a23c149451e25526076b8592669df36c"} Oct 30 06:35:01.861459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:01.861435 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" event={"ID":"7f98765a-50a6-46b3-9931-024c9bd7934c","Type":"ContainerStarted","Data":"f2559b471ffaa91802d141a0c953004fe083e2f2d8aac7a75644e44da93a7478"} Oct 30 06:35:01.922761 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:01.922704 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" podStartSLOduration=1.922689979 podStartE2EDuration="1.922689979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:01.920533968 +0000 UTC m=+2468.429553894" watchObservedRunningTime="2026-10-30 06:35:01.922689979 +0000 UTC m=+2468.431709866" Oct 30 06:35:02.052852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:02.052818 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:02.131041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:02.130949 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 06:35:02.131041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:02.130999 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" podUID="86cc6726-5087-432a-8487-5dd10dfb4c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 06:35:02.200424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:02.200387 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:02.724635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:02.724608 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:02.725846 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:02.725812 6999 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:35:02.725930 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:02.725877 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" podUID="7f98765a-50a6-46b3-9931-024c9bd7934c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:35:02.902230 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:02.902179 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:03.130727 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:03.130692 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body= Oct 30 06:35:03.130897 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:03.130751 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" podUID="86cc6726-5087-432a-8487-5dd10dfb4c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" Oct 30 06:35:03.728671 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:03.728624 6999 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:35:03.729333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:03.729298 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" podUID="7f98765a-50a6-46b3-9931-024c9bd7934c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:35:03.994882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:03.994793 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6"] Oct 30 06:35:04.019520 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.019485 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:04.020307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.020281 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.028666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.028633 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6"] Oct 30 06:35:04.038663 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.038636 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a88d7d\"/\"autores-model-a88d7d-kserve-self-signed-certs\"" Oct 30 06:35:04.038964 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.038946 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a88d7d\"/\"default-dockercfg-nhshm\"" Oct 30 06:35:04.042494 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.042467 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a88d7d\"/\"kube-root-ca.crt\"" Oct 30 06:35:04.042627 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.042565 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a88d7d\"/\"openshift-service-ca.crt\"" Oct 30 06:35:04.117158 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.117126 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-home\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.117382 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.117307 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f236f33a-589f-4f6c-8713-4d6ba3fb0097-tls-certs\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.117477 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.117404 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-dshm\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.117477 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.117428 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-model-cache\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.117668 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.117637 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-tmp-dir\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.117813 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.117793 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5rrl\" (UniqueName: \"kubernetes.io/projected/f236f33a-589f-4f6c-8713-4d6ba3fb0097-kube-api-access-l5rrl\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.150241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.146602 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:35:04.150241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.148269 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:35:04.160245 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.154453 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:35:04.187457 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:35:04.187119 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:35:04.227285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.227243 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-home\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.227648 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.227622 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f236f33a-589f-4f6c-8713-4d6ba3fb0097-tls-certs\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.227819 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.227804 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-dshm\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.227927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.227913 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-model-cache\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.228149 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.228135 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-tmp-dir\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.228373 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.228354 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5rrl\" (UniqueName: \"kubernetes.io/projected/f236f33a-589f-4f6c-8713-4d6ba3fb0097-kube-api-access-l5rrl\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.229444 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.229423 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-home\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.238247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.231729 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-tmp-dir\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.238247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.232265 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-model-cache\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.238247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.234236 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f236f33a-589f-4f6c-8713-4d6ba3fb0097-tls-certs\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.260813 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.257642 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-dshm\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.395746 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.395718 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5rrl\" (UniqueName: \"kubernetes.io/projected/f236f33a-589f-4f6c-8713-4d6ba3fb0097-kube-api-access-l5rrl\") pod \"autores-model-a88d7d-kserve-b75d8579c-qbnv6\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.478231 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.478179 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:04.643367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.642906 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:04.725148 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.725118 6999 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:35:04.725332 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.725172 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" podUID="7f98765a-50a6-46b3-9931-024c9bd7934c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:35:04.975353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:04.975321 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6"] Oct 30 06:35:04.977678 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:35:04.977638 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf236f33a_589f_4f6c_8713_4d6ba3fb0097.slice/crio-063da775c71f4426c2c59db4c1446f23ccdbbbd90a73eaf47aa6376ad2938ddf WatchSource:0}: Error finding container 063da775c71f4426c2c59db4c1446f23ccdbbbd90a73eaf47aa6376ad2938ddf: Status 404 returned error can't find the container with id 063da775c71f4426c2c59db4c1446f23ccdbbbd90a73eaf47aa6376ad2938ddf Oct 30 06:35:05.175178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:05.175143 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:05.724990 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:05.724953 6999 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:35:05.725332 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:05.725023 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" podUID="7f98765a-50a6-46b3-9931-024c9bd7934c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:35:05.912971 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:05.912930 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" event={"ID":"f236f33a-589f-4f6c-8713-4d6ba3fb0097","Type":"ContainerStarted","Data":"571cef6de6cfed43936dc70f7ca3bbe3be8f5c12eb92978e264a4cc9ffa22870"} Oct 30 06:35:05.913162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:05.912987 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" event={"ID":"f236f33a-589f-4f6c-8713-4d6ba3fb0097","Type":"ContainerStarted","Data":"063da775c71f4426c2c59db4c1446f23ccdbbbd90a73eaf47aa6376ad2938ddf"} Oct 30 06:35:05.965544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:05.965480 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" podStartSLOduration=2.965464027 podStartE2EDuration="2.965464027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:05.961674203 +0000 UTC m=+2472.470694091" watchObservedRunningTime="2026-10-30 06:35:05.965464027 +0000 UTC m=+2472.474483913" Oct 30 06:35:06.531547 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:06.531515 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:06.646287 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:06.645320 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:06.724676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:06.724641 6999 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:35:06.724880 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:06.724711 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" podUID="7f98765a-50a6-46b3-9931-024c9bd7934c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:35:07.725437 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:07.725400 6999 patch_prober.go:29] interesting pod/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:35:07.726016 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:07.725459 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" podUID="7f98765a-50a6-46b3-9931-024c9bd7934c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:35:08.468129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:08.468095 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:08.730939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:08.730862 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:08.731356 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:08.731343 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:08.731986 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:08.731967 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:35:10.129647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.129606 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r"] Oct 30 06:35:10.134387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.134367 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.176106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.176071 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r"] Oct 30 06:35:10.235393 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.235363 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-69f3fa-openshift-default-dockercfg-ls84g\"" Oct 30 06:35:10.251594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.251555 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e004f9-f184-4b09-835a-b314650513d4-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.251594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.251594 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e004f9-f184-4b09-835a-b314650513d4-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.251853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.251667 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9cw8\" (UniqueName: \"kubernetes.io/projected/c3e004f9-f184-4b09-835a-b314650513d4-kube-api-access-b9cw8\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.251853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.251742 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.251977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.251920 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.252034 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.252015 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.252090 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.252064 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e004f9-f184-4b09-835a-b314650513d4-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.252156 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.252136 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.252193 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.252171 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.261015 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.260988 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:10.353458 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.353424 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9cw8\" (UniqueName: \"kubernetes.io/projected/c3e004f9-f184-4b09-835a-b314650513d4-kube-api-access-b9cw8\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.353662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.353477 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.353662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.353554 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.353662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.353603 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.353662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.353639 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e004f9-f184-4b09-835a-b314650513d4-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.353869 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.353685 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.353869 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.353710 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.353983 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.353878 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e004f9-f184-4b09-835a-b314650513d4-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.353983 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.353911 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e004f9-f184-4b09-835a-b314650513d4-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.354100 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.354005 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.354100 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.354026 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.354265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.354244 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.354339 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.354253 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.354588 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.354564 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e004f9-f184-4b09-835a-b314650513d4-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.356124 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.356100 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.356559 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.356536 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e004f9-f184-4b09-835a-b314650513d4-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.378996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.378964 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9cw8\" (UniqueName: \"kubernetes.io/projected/c3e004f9-f184-4b09-835a-b314650513d4-kube-api-access-b9cw8\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.379306 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.379283 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e004f9-f184-4b09-835a-b314650513d4-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.462411 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.462328 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:10.789549 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.789522 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r"] Oct 30 06:35:10.790639 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:35:10.790612 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e004f9_f184_4b09_835a_b314650513d4.slice/crio-7de2e0ea2ca9e89f25f3e11739a8d0b4acaf9e6bb6c8bf881d44118121c60379 WatchSource:0}: Error finding container 7de2e0ea2ca9e89f25f3e11739a8d0b4acaf9e6bb6c8bf881d44118121c60379: Status 404 returned error can't find the container with id 7de2e0ea2ca9e89f25f3e11739a8d0b4acaf9e6bb6c8bf881d44118121c60379 Oct 30 06:35:10.793686 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.793652 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:10.793763 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.793720 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:10.793763 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.793751 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:10.954600 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.954495 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" event={"ID":"c3e004f9-f184-4b09-835a-b314650513d4","Type":"ContainerStarted","Data":"dcd27fccf95c79d87b5b36d4046ae1f19e8293551800b155b0c77fc718e6cc5f"} Oct 30 06:35:10.954600 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:10.954565 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" event={"ID":"c3e004f9-f184-4b09-835a-b314650513d4","Type":"ContainerStarted","Data":"7de2e0ea2ca9e89f25f3e11739a8d0b4acaf9e6bb6c8bf881d44118121c60379"} Oct 30 06:35:11.463594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:11.463556 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:11.465452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:11.465429 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:35:11.465602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:11.465473 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" podUID="c3e004f9-f184-4b09-835a-b314650513d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 06:35:12.290135 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:12.290099 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:12.464393 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:12.463793 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:35:12.464393 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:12.463877 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" podUID="c3e004f9-f184-4b09-835a-b314650513d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 06:35:13.463099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:13.463054 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:35:13.463321 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:13.463126 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" podUID="c3e004f9-f184-4b09-835a-b314650513d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 06:35:14.217977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:14.217940 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:14.462890 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:14.462853 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:35:14.463129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:14.462920 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" podUID="c3e004f9-f184-4b09-835a-b314650513d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 06:35:14.647821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:14.647791 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:14.675628 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:14.675593 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:14.718566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:14.718502 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" podStartSLOduration=4.718484964 podStartE2EDuration="4.718484964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:11.004444113 +0000 UTC m=+2477.513464002" watchObservedRunningTime="2026-10-30 06:35:14.718484964 +0000 UTC m=+2481.227504851" Oct 30 06:35:15.011273 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:15.011157 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:15.462789 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:15.462754 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:35:15.463276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:15.462811 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" podUID="c3e004f9-f184-4b09-835a-b314650513d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 06:35:16.257794 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:16.257758 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:16.462970 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:16.462929 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 06:35:16.463565 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:16.463009 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" podUID="c3e004f9-f184-4b09-835a-b314650513d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 06:35:17.468583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:17.468547 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:17.469049 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:17.469021 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:17.469777 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:17.469758 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:35:18.248307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:18.248275 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:20.101706 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:20.101675 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:21.811684 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:21.811653 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:23.662591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:23.662552 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:25.569936 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:25.569898 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:25.736120 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:25.736082 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:27.456854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:27.456790 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:29.375306 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:29.375192 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:30.241484 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:30.241448 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6"] Oct 30 06:35:30.315784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:30.315750 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:30.596889 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:30.596800 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:31.351102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:31.351065 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:31.787495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:31.787464 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:31.939044 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:31.939009 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:32.099143 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:32.099056 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:32.531815 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:32.531781 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:32.732395 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:32.732364 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:33.261944 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:33.261911 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:33.401030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:33.400980 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:33.561686 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:33.561594 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:34.160672 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:35:34.160630 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:35:34.700663 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:34.700631 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:35.356914 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:35.356823 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:37.085867 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:37.085830 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:38.929114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:38.929082 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:40.648044 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:40.648008 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:40.789685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:40.789653 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:41.285347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:41.285315 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:41.430730 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:41.430690 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:41.962920 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:41.962887 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:42.111909 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:42.111880 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:42.268580 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:42.268505 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:42.562944 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:42.562873 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:42.715408 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:42.715375 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:42.862669 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:42.862639 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:43.307892 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:43.307814 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:43.449310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:43.449270 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:44.523254 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:44.523193 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:44.904057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:44.904019 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:45.278533 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.278439 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" podUID="f236f33a-589f-4f6c-8713-4d6ba3fb0097" containerName="main" containerID="cri-o://571cef6de6cfed43936dc70f7ca3bbe3be8f5c12eb92978e264a4cc9ffa22870" gracePeriod=45 Oct 30 06:35:45.625582 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.625551 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:45.764796 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.764768 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-tmp-dir\") pod \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " Oct 30 06:35:45.764962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.764820 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f236f33a-589f-4f6c-8713-4d6ba3fb0097-tls-certs\" (UniqueName: \"kubernetes.io/secret/f236f33a-589f-4f6c-8713-4d6ba3fb0097-tls-certs\") pod \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " Oct 30 06:35:45.764962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.764840 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-home\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-home\") pod \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " Oct 30 06:35:45.764962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.764939 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-dshm\") pod \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " Oct 30 06:35:45.765097 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.765004 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f236f33a-589f-4f6c-8713-4d6ba3fb0097-kube-api-access-l5rrl\" (UniqueName: \"kubernetes.io/projected/f236f33a-589f-4f6c-8713-4d6ba3fb0097-kube-api-access-l5rrl\") pod \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " Oct 30 06:35:45.765097 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.765060 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-tmp-dir" pod "f236f33a-589f-4f6c-8713-4d6ba3fb0097" (UID: "f236f33a-589f-4f6c-8713-4d6ba3fb0097"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:45.765172 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.765142 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-home" pod "f236f33a-589f-4f6c-8713-4d6ba3fb0097" (UID: "f236f33a-589f-4f6c-8713-4d6ba3fb0097"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:45.765231 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.765170 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-model-cache\") pod \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\" (UID: \"f236f33a-589f-4f6c-8713-4d6ba3fb0097\") " Oct 30 06:35:45.765403 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.765381 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-model-cache" pod "f236f33a-589f-4f6c-8713-4d6ba3fb0097" (UID: "f236f33a-589f-4f6c-8713-4d6ba3fb0097"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:45.765791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.765774 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:35:45.765882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.765793 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:35:45.765882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.765802 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:35:45.767391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.767369 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-dshm" pod "f236f33a-589f-4f6c-8713-4d6ba3fb0097" (UID: "f236f33a-589f-4f6c-8713-4d6ba3fb0097"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:45.767488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.767393 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f236f33a-589f-4f6c-8713-4d6ba3fb0097-tls-certs" pod "f236f33a-589f-4f6c-8713-4d6ba3fb0097" (UID: "f236f33a-589f-4f6c-8713-4d6ba3fb0097"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:35:45.767488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.767399 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f236f33a-589f-4f6c-8713-4d6ba3fb0097-kube-api-access-l5rrl" pod "f236f33a-589f-4f6c-8713-4d6ba3fb0097" (UID: "f236f33a-589f-4f6c-8713-4d6ba3fb0097"). InnerVolumeSpecName "kube-api-access-l5rrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:45.866888 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.866862 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f236f33a-589f-4f6c-8713-4d6ba3fb0097-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:35:45.866888 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.866887 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5rrl\" (UniqueName: \"kubernetes.io/projected/f236f33a-589f-4f6c-8713-4d6ba3fb0097-kube-api-access-l5rrl\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:35:45.867078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:45.866898 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f236f33a-589f-4f6c-8713-4d6ba3fb0097-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:35:46.262258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:46.262146 6999 generic.go:356] "Generic (PLEG): container finished" podID="f236f33a-589f-4f6c-8713-4d6ba3fb0097" containerID="571cef6de6cfed43936dc70f7ca3bbe3be8f5c12eb92978e264a4cc9ffa22870" exitCode=0 Oct 30 06:35:46.262258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:46.262195 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" Oct 30 06:35:46.262467 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:46.262258 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" event={"ID":"f236f33a-589f-4f6c-8713-4d6ba3fb0097","Type":"ContainerDied","Data":"571cef6de6cfed43936dc70f7ca3bbe3be8f5c12eb92978e264a4cc9ffa22870"} Oct 30 06:35:46.262467 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:46.262303 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6" event={"ID":"f236f33a-589f-4f6c-8713-4d6ba3fb0097","Type":"ContainerDied","Data":"063da775c71f4426c2c59db4c1446f23ccdbbbd90a73eaf47aa6376ad2938ddf"} Oct 30 06:35:46.262467 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:46.262319 6999 scope.go:122] "RemoveContainer" containerID="571cef6de6cfed43936dc70f7ca3bbe3be8f5c12eb92978e264a4cc9ffa22870" Oct 30 06:35:46.273460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:46.273439 6999 scope.go:122] "RemoveContainer" containerID="571cef6de6cfed43936dc70f7ca3bbe3be8f5c12eb92978e264a4cc9ffa22870" Oct 30 06:35:46.273736 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:35:46.273704 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"571cef6de6cfed43936dc70f7ca3bbe3be8f5c12eb92978e264a4cc9ffa22870\": container with ID starting with 571cef6de6cfed43936dc70f7ca3bbe3be8f5c12eb92978e264a4cc9ffa22870 not found: ID does not exist" containerID="571cef6de6cfed43936dc70f7ca3bbe3be8f5c12eb92978e264a4cc9ffa22870" Oct 30 06:35:46.273858 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:46.273744 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"571cef6de6cfed43936dc70f7ca3bbe3be8f5c12eb92978e264a4cc9ffa22870"} err="failed to get container status \"571cef6de6cfed43936dc70f7ca3bbe3be8f5c12eb92978e264a4cc9ffa22870\": rpc error: code = NotFound desc = could not find container \"571cef6de6cfed43936dc70f7ca3bbe3be8f5c12eb92978e264a4cc9ffa22870\": container with ID starting with 571cef6de6cfed43936dc70f7ca3bbe3be8f5c12eb92978e264a4cc9ffa22870 not found: ID does not exist" Oct 30 06:35:46.319679 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:46.319654 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6"] Oct 30 06:35:46.331080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:46.331053 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-a88d7d/autores-model-a88d7d-kserve-b75d8579c-qbnv6"] Oct 30 06:35:46.350196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:46.350171 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:47.610828 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:47.610796 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f236f33a-589f-4f6c-8713-4d6ba3fb0097" path="/var/lib/kubelet/pods/f236f33a-589f-4f6c-8713-4d6ba3fb0097/volumes" Oct 30 06:35:48.147371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:48.147341 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:50.010391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:50.010354 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:51.816482 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:51.816448 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:53.616584 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:53.616552 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9"] Oct 30 06:35:53.617344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:53.616761 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" podUID="2c2e77f4-eed7-46c9-b1cf-c02f7863431c" containerName="istio-proxy" containerID="cri-o://be6c9bc98adc3320234f6585a388301a4d356ac57ee98a45009ce20b1b0fb1e8" gracePeriod=30 Oct 30 06:35:53.652409 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:53.652378 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:55.267817 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.267775 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9"] Oct 30 06:35:55.268735 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.268713 6999 cpu_manager.go:414] "containerMap: removing container" podUID="f236f33a-589f-4f6c-8713-4d6ba3fb0097" containerName="main" Oct 30 06:35:55.268854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.268736 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f236f33a-589f-4f6c-8713-4d6ba3fb0097" containerName="main" Oct 30 06:35:55.268932 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.268918 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="f236f33a-589f-4f6c-8713-4d6ba3fb0097" containerName="main" Oct 30 06:35:55.272993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.272971 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.309615 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.309589 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9"] Oct 30 06:35:55.323579 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.323554 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.323774 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.323599 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.323774 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.323689 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.323888 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.323862 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.323953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.323889 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.323953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.323910 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.324058 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.324012 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.324119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.324067 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvkgm\" (UniqueName: \"kubernetes.io/projected/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-kube-api-access-cvkgm\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.324174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.324160 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.425882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.425848 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.426038 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.425892 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.426038 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.426018 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.426159 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.426142 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.426257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.426188 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.426257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.426202 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.426342 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.426258 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.426342 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.426304 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.426342 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.426337 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvkgm\" (UniqueName: \"kubernetes.io/projected/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-kube-api-access-cvkgm\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.426503 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.426400 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.426503 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.426430 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.426618 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.426521 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.426678 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.426645 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.427020 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.426996 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.428660 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.428639 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.428997 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.428973 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.456296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.456265 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:55.458491 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.458455 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.461623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.461597 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvkgm\" (UniqueName: \"kubernetes.io/projected/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-kube-api-access-cvkgm\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.595495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.595404 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:55.922495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.922466 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9"] Oct 30 06:35:55.923316 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:35:55.923288 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3934fa5_c92d_4efd_b675_e9764ab0fcfa.slice/crio-9d8f492ab96026ef5f521527935f78c2771cb36afc05906fdbbe496443d5641f WatchSource:0}: Error finding container 9d8f492ab96026ef5f521527935f78c2771cb36afc05906fdbbe496443d5641f: Status 404 returned error can't find the container with id 9d8f492ab96026ef5f521527935f78c2771cb36afc05906fdbbe496443d5641f Oct 30 06:35:55.926113 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.926076 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:55.926203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.926147 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:55.926203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:55.926186 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:35:56.337558 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:56.337469 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" event={"ID":"c3934fa5-c92d-4efd-b675-e9764ab0fcfa","Type":"ContainerStarted","Data":"714ae399d71b09d70ec1fcf5f637b14425595ff8f0621c56baed7f9e7ba64d20"} Oct 30 06:35:56.337558 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:56.337529 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" event={"ID":"c3934fa5-c92d-4efd-b675-e9764ab0fcfa","Type":"ContainerStarted","Data":"9d8f492ab96026ef5f521527935f78c2771cb36afc05906fdbbe496443d5641f"} Oct 30 06:35:56.384079 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:56.384028 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" podStartSLOduration=1.384011262 podStartE2EDuration="1.384011262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:56.382279571 +0000 UTC m=+2522.891299459" watchObservedRunningTime="2026-10-30 06:35:56.384011262 +0000 UTC m=+2522.893031148" Oct 30 06:35:56.595980 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:56.595907 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:35:56.597290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:56.597268 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 06:35:56.597409 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:56.597317 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" podUID="c3934fa5-c92d-4efd-b675-e9764ab0fcfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 06:35:57.270890 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:57.270852 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:57.596113 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:57.596030 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 06:35:57.596113 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:57.596083 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" podUID="c3934fa5-c92d-4efd-b675-e9764ab0fcfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 06:35:58.596647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:58.596609 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 06:35:58.597066 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:58.596660 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" podUID="c3934fa5-c92d-4efd-b675-e9764ab0fcfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 06:35:58.972644 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:58.972619 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:35:59.104309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.104276 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-token\" (UniqueName: \"kubernetes.io/projected/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-token\") pod \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " Oct 30 06:35:59.104509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.104328 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-data\") pod \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " Oct 30 06:35:59.104509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.104371 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-credential-socket\") pod \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " Oct 30 06:35:59.104509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.104467 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-workload-socket\") pod \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " Oct 30 06:35:59.104690 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.104529 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-kube-api-access-qw4d2\" (UniqueName: \"kubernetes.io/projected/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-kube-api-access-qw4d2\") pod \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " Oct 30 06:35:59.104690 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.104639 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-podinfo\") pod \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " Oct 30 06:35:59.104690 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.104665 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istiod-ca-cert\") pod \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " Oct 30 06:35:59.104863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.104702 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-data" pod "2c2e77f4-eed7-46c9-b1cf-c02f7863431c" (UID: "2c2e77f4-eed7-46c9-b1cf-c02f7863431c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:59.104863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.104749 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-workload-socket" pod "2c2e77f4-eed7-46c9-b1cf-c02f7863431c" (UID: "2c2e77f4-eed7-46c9-b1cf-c02f7863431c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:59.104863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.104772 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-envoy\") pod \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " Oct 30 06:35:59.105029 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.104903 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-workload-certs\") pod \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\" (UID: \"2c2e77f4-eed7-46c9-b1cf-c02f7863431c\") " Oct 30 06:35:59.105099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.105069 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istiod-ca-cert" pod "2c2e77f4-eed7-46c9-b1cf-c02f7863431c" (UID: "2c2e77f4-eed7-46c9-b1cf-c02f7863431c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:35:59.105240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.105186 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-credential-socket" pod "2c2e77f4-eed7-46c9-b1cf-c02f7863431c" (UID: "2c2e77f4-eed7-46c9-b1cf-c02f7863431c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:59.105667 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.105636 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-workload-certs" pod "2c2e77f4-eed7-46c9-b1cf-c02f7863431c" (UID: "2c2e77f4-eed7-46c9-b1cf-c02f7863431c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:59.105782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.105716 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.105782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.105741 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.105782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.105758 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.105782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.105772 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.107308 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.107280 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-envoy" pod "2c2e77f4-eed7-46c9-b1cf-c02f7863431c" (UID: "2c2e77f4-eed7-46c9-b1cf-c02f7863431c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:59.108025 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.107989 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-token" pod "2c2e77f4-eed7-46c9-b1cf-c02f7863431c" (UID: "2c2e77f4-eed7-46c9-b1cf-c02f7863431c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:59.109334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.109311 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-podinfo" pod "2c2e77f4-eed7-46c9-b1cf-c02f7863431c" (UID: "2c2e77f4-eed7-46c9-b1cf-c02f7863431c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:35:59.109495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.109473 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-kube-api-access-qw4d2" pod "2c2e77f4-eed7-46c9-b1cf-c02f7863431c" (UID: "2c2e77f4-eed7-46c9-b1cf-c02f7863431c"). InnerVolumeSpecName "kube-api-access-qw4d2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:59.207762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.207681 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.207762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.207720 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw4d2\" (UniqueName: \"kubernetes.io/projected/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-kube-api-access-qw4d2\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.207762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.207730 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.207762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.207739 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.207762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.207748 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c2e77f4-eed7-46c9-b1cf-c02f7863431c-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:35:59.255243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.252894 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:35:59.388332 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.388293 6999 generic.go:356] "Generic (PLEG): container finished" podID="2c2e77f4-eed7-46c9-b1cf-c02f7863431c" containerID="be6c9bc98adc3320234f6585a388301a4d356ac57ee98a45009ce20b1b0fb1e8" exitCode=0 Oct 30 06:35:59.388332 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.388325 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" event={"ID":"2c2e77f4-eed7-46c9-b1cf-c02f7863431c","Type":"ContainerDied","Data":"be6c9bc98adc3320234f6585a388301a4d356ac57ee98a45009ce20b1b0fb1e8"} Oct 30 06:35:59.388608 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.388348 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" Oct 30 06:35:59.388608 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.388379 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9" event={"ID":"2c2e77f4-eed7-46c9-b1cf-c02f7863431c","Type":"ContainerDied","Data":"ce01d2adffdec97c0f42b7a6c302b2f390bcf7b9ac01c00f73d9239dd40456a1"} Oct 30 06:35:59.388608 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.388402 6999 scope.go:122] "RemoveContainer" containerID="be6c9bc98adc3320234f6585a388301a4d356ac57ee98a45009ce20b1b0fb1e8" Oct 30 06:35:59.400351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.400332 6999 scope.go:122] "RemoveContainer" containerID="be6c9bc98adc3320234f6585a388301a4d356ac57ee98a45009ce20b1b0fb1e8" Oct 30 06:35:59.400602 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:35:59.400582 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6c9bc98adc3320234f6585a388301a4d356ac57ee98a45009ce20b1b0fb1e8\": container with ID starting with be6c9bc98adc3320234f6585a388301a4d356ac57ee98a45009ce20b1b0fb1e8 not found: ID does not exist" containerID="be6c9bc98adc3320234f6585a388301a4d356ac57ee98a45009ce20b1b0fb1e8" Oct 30 06:35:59.400677 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.400613 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6c9bc98adc3320234f6585a388301a4d356ac57ee98a45009ce20b1b0fb1e8"} err="failed to get container status \"be6c9bc98adc3320234f6585a388301a4d356ac57ee98a45009ce20b1b0fb1e8\": rpc error: code = NotFound desc = could not find container \"be6c9bc98adc3320234f6585a388301a4d356ac57ee98a45009ce20b1b0fb1e8\": container with ID starting with be6c9bc98adc3320234f6585a388301a4d356ac57ee98a45009ce20b1b0fb1e8 not found: ID does not exist" Oct 30 06:35:59.462781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.462660 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9"] Oct 30 06:35:59.465605 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.465579 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-qndm9"] Oct 30 06:35:59.596245 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.596203 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 06:35:59.596408 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.596282 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" podUID="c3934fa5-c92d-4efd-b675-e9764ab0fcfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 06:35:59.610229 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:35:59.610186 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c2e77f4-eed7-46c9-b1cf-c02f7863431c" path="/var/lib/kubelet/pods/2c2e77f4-eed7-46c9-b1cf-c02f7863431c/volumes" Oct 30 06:36:00.596006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:00.595972 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 06:36:00.596172 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:00.596026 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" podUID="c3934fa5-c92d-4efd-b675-e9764ab0fcfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 06:36:01.114678 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:01.114645 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:01.596284 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:01.596162 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 06:36:01.596284 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:01.596274 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" podUID="c3934fa5-c92d-4efd-b675-e9764ab0fcfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 06:36:02.600569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:02.600541 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:36:02.601027 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:02.600835 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:36:02.601613 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:02.601596 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:36:02.968887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:02.968854 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:03.119603 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:03.119568 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:04.143234 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:36:04.143129 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:36:04.757181 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:04.757152 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:05.255161 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:05.255100 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:06.611192 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:06.611151 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:08.373432 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:08.373393 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:10.125962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.125923 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:10.301130 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.301070 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96"] Oct 30 06:36:10.301997 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.301978 6999 cpu_manager.go:414] "containerMap: removing container" podUID="2c2e77f4-eed7-46c9-b1cf-c02f7863431c" containerName="istio-proxy" Oct 30 06:36:10.302094 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.301998 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2e77f4-eed7-46c9-b1cf-c02f7863431c" containerName="istio-proxy" Oct 30 06:36:10.302138 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.302121 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2e77f4-eed7-46c9-b1cf-c02f7863431c" containerName="istio-proxy" Oct 30 06:36:10.307639 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.307615 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.339265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.339237 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96"] Oct 30 06:36:10.377740 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.377663 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.377740 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.377714 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.377950 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.377806 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.377950 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.377857 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.377950 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.377946 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.378133 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.377981 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62421fb5-a574-4d99-a8e8-747f202e7e24-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.378133 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.378031 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk4dw\" (UniqueName: \"kubernetes.io/projected/62421fb5-a574-4d99-a8e8-747f202e7e24-kube-api-access-jk4dw\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.378133 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.378048 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.378133 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.378068 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.479657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.479626 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.479821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.479681 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62421fb5-a574-4d99-a8e8-747f202e7e24-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.479821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.479747 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk4dw\" (UniqueName: \"kubernetes.io/projected/62421fb5-a574-4d99-a8e8-747f202e7e24-kube-api-access-jk4dw\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.479821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.479777 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.479821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.479808 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.480028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.480001 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.480084 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.480047 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.480084 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.480062 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-data\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.480084 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.480074 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.480246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.480125 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.480246 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.480181 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-workload-certs\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.480368 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.480257 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-credential-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.480430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.480391 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-workload-socket\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.480530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.480508 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62421fb5-a574-4d99-a8e8-747f202e7e24-istiod-ca-cert\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.482626 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.482602 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-envoy\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.482829 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.482804 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-podinfo\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.502742 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.502713 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-token\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.502904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.502883 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk4dw\" (UniqueName: \"kubernetes.io/projected/62421fb5-a574-4d99-a8e8-747f202e7e24-kube-api-access-jk4dw\") pod \"e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.623413 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.623382 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:10.949259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.949230 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96"] Oct 30 06:36:10.952137 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:36:10.952105 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62421fb5_a574_4d99_a8e8_747f202e7e24.slice/crio-dbfb5a29125b9e2586534d70f01deee0cfaa38a99641152f0468dffb3dcccb93 WatchSource:0}: Error finding container dbfb5a29125b9e2586534d70f01deee0cfaa38a99641152f0468dffb3dcccb93: Status 404 returned error can't find the container with id dbfb5a29125b9e2586534d70f01deee0cfaa38a99641152f0468dffb3dcccb93 Oct 30 06:36:10.955457 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.955412 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:36:10.955546 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.955502 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:36:10.955589 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:10.955543 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:36:11.476090 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:11.476052 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" event={"ID":"62421fb5-a574-4d99-a8e8-747f202e7e24","Type":"ContainerStarted","Data":"4b41a093dc882dcd94773ab6abca39dc4ef7b3149bac586881249b055d9d62be"} Oct 30 06:36:11.476543 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:11.476118 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" event={"ID":"62421fb5-a574-4d99-a8e8-747f202e7e24","Type":"ContainerStarted","Data":"dbfb5a29125b9e2586534d70f01deee0cfaa38a99641152f0468dffb3dcccb93"} Oct 30 06:36:11.521373 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:11.521314 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" podStartSLOduration=1.521296 podStartE2EDuration="1.521296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:11.520452364 +0000 UTC m=+2538.029472263" watchObservedRunningTime="2026-10-30 06:36:11.521296 +0000 UTC m=+2538.030315892" Oct 30 06:36:11.623842 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:11.623813 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:11.857514 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:11.857439 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:12.625848 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:12.625813 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 06:36:12.626372 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:12.625880 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" podUID="62421fb5-a574-4d99-a8e8-747f202e7e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 06:36:13.623862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:13.623825 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 06:36:13.624047 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:13.623888 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" podUID="62421fb5-a574-4d99-a8e8-747f202e7e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 06:36:13.660788 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:13.660694 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:14.624365 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:14.624335 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 06:36:14.624583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:14.624393 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" podUID="62421fb5-a574-4d99-a8e8-747f202e7e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 06:36:15.468601 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:15.468570 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:15.624649 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:15.624612 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 06:36:15.624814 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:15.624674 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" podUID="62421fb5-a574-4d99-a8e8-747f202e7e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 06:36:16.624281 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:16.624250 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 06:36:16.624715 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:16.624303 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" podUID="62421fb5-a574-4d99-a8e8-747f202e7e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 06:36:17.508085 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:17.499767 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:17.660145 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:17.659680 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:17.660688 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:17.660175 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:17.662006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:17.661976 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:36:19.429348 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:19.429302 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:21.362840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:21.362805 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:22.839508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:22.839474 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6"] Oct 30 06:36:22.840048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:22.839795 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" podUID="1bca0367-1f0a-4d33-8bcc-551a7c5d1bda" containerName="maas-api" containerID="cri-o://840032b60be0389171423b434774858499b3e0aa1450d617ba2a0974abd18868" gracePeriod=30 Oct 30 06:36:22.927739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:22.927700 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:23.145518 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.145493 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:36:23.320496 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.320466 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-maas-api-tls\") pod \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " Oct 30 06:36:23.320732 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.320715 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-metrics-certs\") pod \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " Oct 30 06:36:23.320817 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.320778 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-kube-api-access-cj6gl\" (UniqueName: \"kubernetes.io/projected/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-kube-api-access-cj6gl\") pod \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " Oct 30 06:36:23.320876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.320830 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-openshift-service-ca\") pod \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\" (UID: \"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda\") " Oct 30 06:36:23.321542 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.321340 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-openshift-service-ca" pod "1bca0367-1f0a-4d33-8bcc-551a7c5d1bda" (UID: "1bca0367-1f0a-4d33-8bcc-551a7c5d1bda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:36:23.321542 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.321519 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:36:23.327964 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.327904 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-maas-api-tls" pod "1bca0367-1f0a-4d33-8bcc-551a7c5d1bda" (UID: "1bca0367-1f0a-4d33-8bcc-551a7c5d1bda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:36:23.327964 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.327934 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-metrics-certs" pod "1bca0367-1f0a-4d33-8bcc-551a7c5d1bda" (UID: "1bca0367-1f0a-4d33-8bcc-551a7c5d1bda"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:36:23.327964 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.327957 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-kube-api-access-cj6gl" pod "1bca0367-1f0a-4d33-8bcc-551a7c5d1bda" (UID: "1bca0367-1f0a-4d33-8bcc-551a7c5d1bda"). InnerVolumeSpecName "kube-api-access-cj6gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:23.445299 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.441436 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:36:23.445299 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.441469 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj6gl\" (UniqueName: \"kubernetes.io/projected/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-kube-api-access-cj6gl\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:36:23.445299 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.441485 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:36:23.551361 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.551322 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:23.569841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.569808 6999 generic.go:356] "Generic (PLEG): container finished" podID="1bca0367-1f0a-4d33-8bcc-551a7c5d1bda" containerID="840032b60be0389171423b434774858499b3e0aa1450d617ba2a0974abd18868" exitCode=0 Oct 30 06:36:23.570018 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.569854 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" Oct 30 06:36:23.570018 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.569879 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" event={"ID":"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda","Type":"ContainerDied","Data":"840032b60be0389171423b434774858499b3e0aa1450d617ba2a0974abd18868"} Oct 30 06:36:23.570018 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.569944 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6" event={"ID":"1bca0367-1f0a-4d33-8bcc-551a7c5d1bda","Type":"ContainerDied","Data":"9a34d68831cb5d47964cea642e48c5eef36e16ee8fe86d5fbf3534282cae81ba"} Oct 30 06:36:23.570018 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.569975 6999 scope.go:122] "RemoveContainer" containerID="840032b60be0389171423b434774858499b3e0aa1450d617ba2a0974abd18868" Oct 30 06:36:23.582909 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.582889 6999 scope.go:122] "RemoveContainer" containerID="840032b60be0389171423b434774858499b3e0aa1450d617ba2a0974abd18868" Oct 30 06:36:23.583198 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:36:23.583171 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"840032b60be0389171423b434774858499b3e0aa1450d617ba2a0974abd18868\": container with ID starting with 840032b60be0389171423b434774858499b3e0aa1450d617ba2a0974abd18868 not found: ID does not exist" containerID="840032b60be0389171423b434774858499b3e0aa1450d617ba2a0974abd18868" Oct 30 06:36:23.583314 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.583206 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840032b60be0389171423b434774858499b3e0aa1450d617ba2a0974abd18868"} err="failed to get container status \"840032b60be0389171423b434774858499b3e0aa1450d617ba2a0974abd18868\": rpc error: code = NotFound desc = could not find container \"840032b60be0389171423b434774858499b3e0aa1450d617ba2a0974abd18868\": container with ID starting with 840032b60be0389171423b434774858499b3e0aa1450d617ba2a0974abd18868 not found: ID does not exist" Oct 30 06:36:23.624178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.624151 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6"] Oct 30 06:36:23.638368 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.638345 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a88d7d-59c96d6cc7-fm7g6"] Oct 30 06:36:23.707463 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:23.707362 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:24.333603 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:24.333576 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:24.784364 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:24.784330 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:24.936430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:24.936396 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:25.610233 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:25.610188 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bca0367-1f0a-4d33-8bcc-551a7c5d1bda" path="/var/lib/kubelet/pods/1bca0367-1f0a-4d33-8bcc-551a7c5d1bda/volumes" Oct 30 06:36:25.703062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:25.703005 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:25.851293 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:25.851257 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:25.995140 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:25.995039 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:27.729882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:27.729826 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:29.540578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:29.540541 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:31.061277 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:31.061246 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:31.201043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:31.201012 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:31.353356 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:31.353268 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:31.398837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:31.398795 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9"] Oct 30 06:36:31.399926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:31.399862 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" podUID="7f98765a-50a6-46b3-9931-024c9bd7934c" containerName="istio-proxy" containerID="cri-o://3befa12c347c86e19ff801063898dc89a23c149451e25526076b8592669df36c" gracePeriod=30 Oct 30 06:36:31.525237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:31.524496 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:33.140316 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:33.140285 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:33.474253 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:33.474142 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:34.076161 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:34.076129 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:34.174905 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:36:34.174864 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:36:34.420562 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:34.420529 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:35.137488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:35.137452 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:35.982962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:35.982931 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:36.129568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.129528 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:36.671841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.671545 6999 generic.go:356] "Generic (PLEG): container finished" podID="7f98765a-50a6-46b3-9931-024c9bd7934c" containerID="3befa12c347c86e19ff801063898dc89a23c149451e25526076b8592669df36c" exitCode=0 Oct 30 06:36:36.671841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.671763 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" event={"ID":"7f98765a-50a6-46b3-9931-024c9bd7934c","Type":"ContainerDied","Data":"3befa12c347c86e19ff801063898dc89a23c149451e25526076b8592669df36c"} Oct 30 06:36:36.672656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.672629 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:36.756545 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.756522 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:36:36.867010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.866972 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-workload-certs\") pod \"7f98765a-50a6-46b3-9931-024c9bd7934c\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " Oct 30 06:36:36.867175 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.867028 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-credential-socket\") pod \"7f98765a-50a6-46b3-9931-024c9bd7934c\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " Oct 30 06:36:36.867175 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.867102 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-token\" (UniqueName: \"kubernetes.io/projected/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-token\") pod \"7f98765a-50a6-46b3-9931-024c9bd7934c\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " Oct 30 06:36:36.867341 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.867181 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-data\") pod \"7f98765a-50a6-46b3-9931-024c9bd7934c\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " Oct 30 06:36:36.867411 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.867332 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-workload-certs" pod "7f98765a-50a6-46b3-9931-024c9bd7934c" (UID: "7f98765a-50a6-46b3-9931-024c9bd7934c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:36.867411 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.867351 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-workload-socket\") pod \"7f98765a-50a6-46b3-9931-024c9bd7934c\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " Oct 30 06:36:36.867528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.867431 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f98765a-50a6-46b3-9931-024c9bd7934c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f98765a-50a6-46b3-9931-024c9bd7934c-istiod-ca-cert\") pod \"7f98765a-50a6-46b3-9931-024c9bd7934c\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " Oct 30 06:36:36.867528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.867448 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-credential-socket" pod "7f98765a-50a6-46b3-9931-024c9bd7934c" (UID: "7f98765a-50a6-46b3-9931-024c9bd7934c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:36.867528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.867469 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f98765a-50a6-46b3-9931-024c9bd7934c-kube-api-access-f6hbv\" (UniqueName: \"kubernetes.io/projected/7f98765a-50a6-46b3-9931-024c9bd7934c-kube-api-access-f6hbv\") pod \"7f98765a-50a6-46b3-9931-024c9bd7934c\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " Oct 30 06:36:36.867528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.867504 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-podinfo\") pod \"7f98765a-50a6-46b3-9931-024c9bd7934c\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " Oct 30 06:36:36.867744 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.867570 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-data" pod "7f98765a-50a6-46b3-9931-024c9bd7934c" (UID: "7f98765a-50a6-46b3-9931-024c9bd7934c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:36.867744 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.867669 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-workload-socket" pod "7f98765a-50a6-46b3-9931-024c9bd7934c" (UID: "7f98765a-50a6-46b3-9931-024c9bd7934c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:36.867744 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.867686 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-envoy\") pod \"7f98765a-50a6-46b3-9931-024c9bd7934c\" (UID: \"7f98765a-50a6-46b3-9931-024c9bd7934c\") " Oct 30 06:36:36.867958 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.867905 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f98765a-50a6-46b3-9931-024c9bd7934c-istiod-ca-cert" pod "7f98765a-50a6-46b3-9931-024c9bd7934c" (UID: "7f98765a-50a6-46b3-9931-024c9bd7934c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:36:36.868441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.868364 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:36:36.868441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.868398 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f98765a-50a6-46b3-9931-024c9bd7934c-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:36:36.868441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.868411 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:36:36.868441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.868425 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:36:36.868441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.868438 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:36:36.870647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.870606 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-podinfo" pod "7f98765a-50a6-46b3-9931-024c9bd7934c" (UID: "7f98765a-50a6-46b3-9931-024c9bd7934c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:36:36.870891 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.870862 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-envoy" pod "7f98765a-50a6-46b3-9931-024c9bd7934c" (UID: "7f98765a-50a6-46b3-9931-024c9bd7934c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:36:36.870987 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.870902 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-token" pod "7f98765a-50a6-46b3-9931-024c9bd7934c" (UID: "7f98765a-50a6-46b3-9931-024c9bd7934c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:36.870987 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.870944 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f98765a-50a6-46b3-9931-024c9bd7934c-kube-api-access-f6hbv" pod "7f98765a-50a6-46b3-9931-024c9bd7934c" (UID: "7f98765a-50a6-46b3-9931-024c9bd7934c"). InnerVolumeSpecName "kube-api-access-f6hbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:36.969947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.969910 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:36:36.969947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.969948 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6hbv\" (UniqueName: \"kubernetes.io/projected/7f98765a-50a6-46b3-9931-024c9bd7934c-kube-api-access-f6hbv\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:36:36.970199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.969963 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:36:36.970199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:36.969975 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f98765a-50a6-46b3-9931-024c9bd7934c-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:36:37.128067 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:37.127928 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:37.685182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:37.685152 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" event={"ID":"7f98765a-50a6-46b3-9931-024c9bd7934c","Type":"ContainerDied","Data":"f2559b471ffaa91802d141a0c953004fe083e2f2d8aac7a75644e44da93a7478"} Oct 30 06:36:37.685182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:37.685181 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9" Oct 30 06:36:37.685450 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:37.685233 6999 scope.go:122] "RemoveContainer" containerID="3befa12c347c86e19ff801063898dc89a23c149451e25526076b8592669df36c" Oct 30 06:36:37.770236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:37.768576 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9"] Oct 30 06:36:37.783321 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:37.783282 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a88d7d-openshift-default-5bf49799d7-jlrk9"] Oct 30 06:36:37.801951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:37.801925 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:37.945889 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:37.945812 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:38.501521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:38.501491 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:38.682472 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:38.682438 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:38.831503 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:38.831420 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:39.233556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:39.233523 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:39.610139 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:39.610110 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f98765a-50a6-46b3-9931-024c9bd7934c" path="/var/lib/kubelet/pods/7f98765a-50a6-46b3-9931-024c9bd7934c/volumes" Oct 30 06:36:40.377896 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:40.377863 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:40.517940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:40.517908 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:41.125361 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:41.125327 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:42.996736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:42.996575 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:43.667542 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:43.667503 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:43.927432 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:43.927313 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:44.076555 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:44.076522 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:44.918960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:44.918926 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:45.705368 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:45.705334 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:45.856400 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:45.856371 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:45.996635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:45.996531 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:46.721258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:46.721228 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:47.367322 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:47.367284 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:47.513263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:47.513202 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:48.713160 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:48.713122 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:50.433440 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:50.433410 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:50.915315 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:50.915282 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:51.056470 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:51.056438 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:51.501296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:51.501264 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:51.659614 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:51.659547 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:52.502605 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:52.502570 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:52.788324 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:52.788245 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:53.771754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:53.771721 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:53.937104 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:53.937069 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:54.276007 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:54.275971 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:54.420909 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:54.420878 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:54.571929 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:54.571821 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:55.507265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:55.507240 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:56.134856 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:56.134824 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:57.870183 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:57.870143 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:36:59.616926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:36:59.616891 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:01.554795 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:01.554749 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:03.195484 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:03.195446 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:03.352071 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:03.352036 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:04.143706 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:37:04.143664 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:37:05.138360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:05.138330 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:05.434629 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:05.434536 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:06.922095 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:06.922058 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:08.656338 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:08.656306 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:10.401538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:10.401505 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:12.233324 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:12.233252 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:14.310574 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:14.310491 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:16.213369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:16.213338 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:17.005920 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:17.005883 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:17.150571 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:17.150529 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:18.260772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:18.260734 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:18.485892 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:18.485848 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:18.654837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:18.654805 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:18.792660 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:18.792619 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:20.000040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.000005 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:20.386675 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.386638 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj"] Oct 30 06:37:20.387799 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.387774 6999 cpu_manager.go:414] "containerMap: removing container" podUID="1bca0367-1f0a-4d33-8bcc-551a7c5d1bda" containerName="maas-api" Oct 30 06:37:20.387915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.387802 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bca0367-1f0a-4d33-8bcc-551a7c5d1bda" containerName="maas-api" Oct 30 06:37:20.387915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.387833 6999 cpu_manager.go:414] "containerMap: removing container" podUID="7f98765a-50a6-46b3-9931-024c9bd7934c" containerName="istio-proxy" Oct 30 06:37:20.387915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.387841 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f98765a-50a6-46b3-9931-024c9bd7934c" containerName="istio-proxy" Oct 30 06:37:20.388079 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.388046 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bca0367-1f0a-4d33-8bcc-551a7c5d1bda" containerName="maas-api" Oct 30 06:37:20.388079 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.388061 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f98765a-50a6-46b3-9931-024c9bd7934c" containerName="istio-proxy" Oct 30 06:37:20.392571 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.392549 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:20.421174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.421131 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj"] Oct 30 06:37:20.421748 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.421725 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-88acce-serving-cert\"" Oct 30 06:37:20.507802 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.507768 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:20.517321 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.517286 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead1e0b2-5bb1-451d-865d-2fc40e243987-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:20.517512 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.517448 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead1e0b2-5bb1-451d-865d-2fc40e243987-maas-api-tls\") pod \"maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:20.517571 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.517497 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwz7k\" (UniqueName: \"kubernetes.io/projected/ead1e0b2-5bb1-451d-865d-2fc40e243987-kube-api-access-rwz7k\") pod \"maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:20.517725 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.517709 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead1e0b2-5bb1-451d-865d-2fc40e243987-metrics-certs\") pod \"maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:20.622615 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.622568 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead1e0b2-5bb1-451d-865d-2fc40e243987-metrics-certs\") pod \"maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:20.623500 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.623459 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead1e0b2-5bb1-451d-865d-2fc40e243987-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:20.623646 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.623619 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead1e0b2-5bb1-451d-865d-2fc40e243987-maas-api-tls\") pod \"maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:20.623726 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.623679 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwz7k\" (UniqueName: \"kubernetes.io/projected/ead1e0b2-5bb1-451d-865d-2fc40e243987-kube-api-access-rwz7k\") pod \"maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:20.625692 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.625672 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead1e0b2-5bb1-451d-865d-2fc40e243987-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:20.650825 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.650754 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead1e0b2-5bb1-451d-865d-2fc40e243987-maas-api-tls\") pod \"maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:20.651574 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.651550 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead1e0b2-5bb1-451d-865d-2fc40e243987-metrics-certs\") pod \"maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:20.658888 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.658836 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwz7k\" (UniqueName: \"kubernetes.io/projected/ead1e0b2-5bb1-451d-865d-2fc40e243987-kube-api-access-rwz7k\") pod \"maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:20.661333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.661309 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:20.718798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.718763 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:20.803254 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:20.802998 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:21.069839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:21.069815 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj"] Oct 30 06:37:21.070676 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:37:21.070652 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead1e0b2_5bb1_451d_865d_2fc40e243987.slice/crio-d21cc36fe1658ce0f9add4ab420a33724ffd2494ae1a7a03d776d818a5e46194 WatchSource:0}: Error finding container d21cc36fe1658ce0f9add4ab420a33724ffd2494ae1a7a03d776d818a5e46194: Status 404 returned error can't find the container with id d21cc36fe1658ce0f9add4ab420a33724ffd2494ae1a7a03d776d818a5e46194 Oct 30 06:37:21.211630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:21.211580 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" event={"ID":"ead1e0b2-5bb1-451d-865d-2fc40e243987","Type":"ContainerStarted","Data":"d21cc36fe1658ce0f9add4ab420a33724ffd2494ae1a7a03d776d818a5e46194"} Oct 30 06:37:22.222699 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:22.222666 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" event={"ID":"ead1e0b2-5bb1-451d-865d-2fc40e243987","Type":"ContainerStarted","Data":"c8732042a2c3397b07cb3b34abb1a7da87412211a2fe77f57813e9df7194a2ff"} Oct 30 06:37:22.223315 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:22.223288 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:22.266775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:22.266722 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" podStartSLOduration=1.857560492 podStartE2EDuration="2.266706535s" totalImagesPullingTime="409.146043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:37:22.265889979 +0000 UTC m=+2608.774909868" watchObservedRunningTime="2026-10-30 06:37:22.266706535 +0000 UTC m=+2608.775726421" Oct 30 06:37:22.432444 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:22.432414 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:24.229593 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:24.229561 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:24.612193 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:24.612160 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:24.764237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:24.759378 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:26.267791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:26.267758 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:26.564612 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:26.564535 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:26.895854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:26.895822 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:27.064732 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:27.064700 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:27.555275 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:27.555243 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:28.230283 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:28.230244 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:28.235038 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:28.235000 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:37:29.926352 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:29.926312 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:31.736451 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:31.736423 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:32.920068 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:32.920031 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:33.061993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:33.061963 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:33.558081 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:33.557992 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:34.143160 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:37:34.143126 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:37:35.436792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:35.436761 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:37.314879 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:37.314845 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:39.197954 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:39.197916 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:41.058710 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:41.058674 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:42.842690 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:42.842659 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:44.630441 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:44.630408 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:45.722542 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:45.722509 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:45.871495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:45.871465 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:46.395040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:46.394983 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:46.620852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:46.620821 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:48.428473 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:48.428433 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:50.131462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:50.131428 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:51.935087 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:51.935055 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:52.698950 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:52.698922 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:52.838033 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:52.837999 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:52.974104 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:52.974014 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:53.702235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:53.702189 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:55.427745 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:55.427712 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:57.267294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:57.267264 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:59.051786 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:59.051723 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:59.275001 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:59.274973 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:37:59.422800 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:37:59.422773 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:00.259700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:00.259662 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:00.400897 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:00.400869 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:00.995798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:00.995770 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:01.843988 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:01.843955 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:02.837411 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:02.837375 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:03.322320 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:03.322288 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:04.143964 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:38:04.143920 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:38:04.724866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:04.724835 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:05.560554 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:05.560521 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:05.729923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:05.729890 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:06.579656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:06.579621 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:07.121300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:07.121268 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:07.602012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:07.601928 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:08.025183 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:08.025131 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:08.415230 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:08.415176 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:10.230626 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:10.230586 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:11.621307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:11.621262 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:11.769498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:11.769469 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:12.200371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:12.200337 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:13.318171 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:13.318135 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:13.460381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:13.460352 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:14.119882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:14.119847 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:15.004692 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:15.004655 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:15.941646 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:15.941613 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:17.801162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:17.801129 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:19.813004 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:19.812966 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:21.637398 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:21.637366 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:23.340354 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:23.340324 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:25.122374 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:25.122338 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:26.943007 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:26.942976 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:28.684871 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:28.684842 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:30.479820 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:30.479785 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:32.260456 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:32.260431 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:32.521464 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:32.521367 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:32.656200 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:32.656167 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:32.856536 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:32.855045 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:33.247672 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:33.247640 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:33.382885 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:33.382851 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:33.555968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:33.555886 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:33.706817 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:33.706788 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:33.859769 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:33.859738 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:34.002052 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:34.002024 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:34.143924 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:38:34.143837 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:38:34.262988 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:34.262961 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:34.638657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:34.638627 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:34.775637 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:34.775608 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:35.742548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:35.742516 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:35.947124 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:35.947089 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:36.104642 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:36.104555 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:36.465746 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:36.465720 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:38.324172 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:38.324137 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:39.487115 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:39.487083 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:39.629509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:39.629476 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:39.847958 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:39.847879 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:40.305553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:40.305524 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:40.446199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:40.445953 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:40.625032 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:40.624901 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:40.782972 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:40.782935 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:40.933415 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:40.933321 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:41.114503 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:41.114468 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:41.756782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:41.756740 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:41.898411 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:41.898375 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:42.388655 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:42.388605 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:42.867586 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:42.867540 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:43.017777 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:43.017738 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:43.170915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:43.170815 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:44.352979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:44.352945 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:46.171951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:46.171914 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:47.934687 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:47.934647 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:49.727003 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:49.726969 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:51.508835 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:51.508800 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:53.263960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:53.263920 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:54.874692 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:54.874660 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52"] Oct 30 06:38:54.879278 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:54.879261 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:54.904063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:54.904037 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-c44730-serving-cert\"" Oct 30 06:38:54.911143 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:54.911120 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52"] Oct 30 06:38:54.993748 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:54.993710 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:55.038869 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.038837 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-476bm\" (UniqueName: \"kubernetes.io/projected/db9631b2-8f37-4465-83e0-6c67404aeb5b-kube-api-access-476bm\") pod \"maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:55.039043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.038886 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9631b2-8f37-4465-83e0-6c67404aeb5b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:55.039043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.039010 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-metrics-certs\") pod \"maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:55.039043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.039037 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:55.140307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.140184 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-476bm\" (UniqueName: \"kubernetes.io/projected/db9631b2-8f37-4465-83e0-6c67404aeb5b-kube-api-access-476bm\") pod \"maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:55.140307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.140282 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9631b2-8f37-4465-83e0-6c67404aeb5b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:55.140553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.140348 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-metrics-certs\") pod \"maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:55.140553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.140367 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:55.140667 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:38:55.140557 6999 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-serving-cert: secret "maas-api-e2e-extmod-b-c44730-serving-cert" not found Oct 30 06:38:55.140667 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:38:55.140621 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-maas-api-tls podName:db9631b2-8f37-4465-83e0-6c67404aeb5b nodeName:}" failed. No retries permitted until 2026-10-30 06:38:55.640604084 +0000 UTC m=+2702.149623967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-maas-api-tls") pod "maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" (UID: "db9631b2-8f37-4465-83e0-6c67404aeb5b") : secret "maas-api-e2e-extmod-b-c44730-serving-cert" not found Oct 30 06:38:55.141014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.140995 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9631b2-8f37-4465-83e0-6c67404aeb5b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:55.143330 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.143305 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-metrics-certs\") pod \"maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:55.144556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.144529 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:55.170451 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.170416 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-476bm\" (UniqueName: \"kubernetes.io/projected/db9631b2-8f37-4465-83e0-6c67404aeb5b-kube-api-access-476bm\") pod \"maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:55.289517 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.289487 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:55.646866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.646836 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:55.649532 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.649507 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:55.799644 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:55.799612 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:56.142589 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:56.142561 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52"] Oct 30 06:38:56.143492 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:38:56.143466 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9631b2_8f37_4465_83e0_6c67404aeb5b.slice/crio-c549e6ba061d565bbf89ed6f5dd2ab3381094b7da1cccb9be299857dee1e5d35 WatchSource:0}: Error finding container c549e6ba061d565bbf89ed6f5dd2ab3381094b7da1cccb9be299857dee1e5d35: Status 404 returned error can't find the container with id c549e6ba061d565bbf89ed6f5dd2ab3381094b7da1cccb9be299857dee1e5d35 Oct 30 06:38:56.145399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:56.145379 6999 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:38:56.906615 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:56.906577 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:57.045490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.045453 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" event={"ID":"db9631b2-8f37-4465-83e0-6c67404aeb5b","Type":"ContainerStarted","Data":"f72f9b32410e2fc5b5c385e4fe208f26f4f586f8c50c4336bdb36162f84e9fab"} Oct 30 06:38:57.045692 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.045544 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:38:57.045692 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.045566 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" event={"ID":"db9631b2-8f37-4465-83e0-6c67404aeb5b","Type":"ContainerStarted","Data":"c549e6ba061d565bbf89ed6f5dd2ab3381094b7da1cccb9be299857dee1e5d35"} Oct 30 06:38:57.089758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.089709 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" podStartSLOduration=2.710338247 podStartE2EDuration="3.08969558s" totalImagesPullingTime="379.357333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:57.087426707 +0000 UTC m=+2703.596446596" watchObservedRunningTime="2026-10-30 06:38:57.08969558 +0000 UTC m=+2703.598715466" Oct 30 06:38:57.432127 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.432087 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s"] Oct 30 06:38:57.437112 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.437091 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.460171 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.460143 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s"] Oct 30 06:38:57.467331 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.467302 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dbg4t\"" Oct 30 06:38:57.467487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.467377 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 06:38:57.467487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.467437 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 06:38:57.555040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.555013 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 06:38:57.574868 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.574823 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm9bc\" (UniqueName: \"kubernetes.io/projected/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-kube-api-access-bm9bc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.574992 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.574938 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.575044 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.575007 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.575104 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.575054 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.575143 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.575121 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.575179 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.575172 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.575359 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.575339 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.694304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.690330 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.694304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.690459 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.694304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.690521 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm9bc\" (UniqueName: \"kubernetes.io/projected/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-kube-api-access-bm9bc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.694304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.690584 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.694304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.690636 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.694304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.690694 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.694304 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.690761 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.706240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.705789 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.716552 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.716522 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.717642 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.717616 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.717838 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.717624 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.721542 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.721517 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.727718 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.727375 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm9bc\" (UniqueName: \"kubernetes.io/projected/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-kube-api-access-bm9bc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.730230 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.730172 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7kz9s\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:57.756264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:57.752497 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:38:58.124589 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:58.124556 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s"] Oct 30 06:38:58.130056 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:38:58.130020 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8eee62_2ff4_4caa_a795_79ae5e1ffcd2.slice/crio-38c4cd712d7580894e724da6b032877a17ff3fa1ceb77c336429201355b452da WatchSource:0}: Error finding container 38c4cd712d7580894e724da6b032877a17ff3fa1ceb77c336429201355b452da: Status 404 returned error can't find the container with id 38c4cd712d7580894e724da6b032877a17ff3fa1ceb77c336429201355b452da Oct 30 06:38:58.916850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:58.916815 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:38:59.069477 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:38:59.069438 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" event={"ID":"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2","Type":"ContainerStarted","Data":"38c4cd712d7580894e724da6b032877a17ff3fa1ceb77c336429201355b452da"} Oct 30 06:39:00.957643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:00.956968 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:01.124542 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.124507 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c"] Oct 30 06:39:01.128858 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.128829 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.156351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.156316 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.156532 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.156373 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.156532 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.156417 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.156532 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.156522 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.164599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.156587 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.164599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.156746 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.164599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.156779 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d15937-bbc6-440d-ba22-f7989f1e774d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.164599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.156807 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm9qj\" (UniqueName: \"kubernetes.io/projected/85d15937-bbc6-440d-ba22-f7989f1e774d-kube-api-access-pm9qj\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.164599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.156860 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.164599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.163851 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c"] Oct 30 06:39:01.183729 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.183699 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:39:01.183845 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.183756 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:39:01.258585 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.258559 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.258712 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.258594 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d15937-bbc6-440d-ba22-f7989f1e774d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.258712 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.258613 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm9qj\" (UniqueName: \"kubernetes.io/projected/85d15937-bbc6-440d-ba22-f7989f1e774d-kube-api-access-pm9qj\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.258712 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.258664 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.258872 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.258723 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.258929 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.258862 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.259007 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.258928 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.259007 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.259000 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.259121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.259031 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.259121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.259061 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.259121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.259091 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.259121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.259102 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.259404 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.259380 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.259452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.259435 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d15937-bbc6-440d-ba22-f7989f1e774d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.261475 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.261453 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.261715 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.261698 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.282002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.281973 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.282119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.282063 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm9qj\" (UniqueName: \"kubernetes.io/projected/85d15937-bbc6-440d-ba22-f7989f1e774d-kube-api-access-pm9qj\") pod \"maas-default-gateway-openshift-default-58847f9fc5-b9r4c\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.448516 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.448483 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:01.789853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.787765 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c"] Oct 30 06:39:01.806257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.806158 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:39:01.806407 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.806293 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:39:01.806407 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:01.806377 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:39:02.101608 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:02.101511 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" event={"ID":"85d15937-bbc6-440d-ba22-f7989f1e774d","Type":"ContainerStarted","Data":"7b28bb76c6d0083666ebe47272cf92459120bcd7d193d1539fe6cad17563bcf3"} Oct 30 06:39:02.102201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:02.102136 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" event={"ID":"85d15937-bbc6-440d-ba22-f7989f1e774d","Type":"ContainerStarted","Data":"7d4b44a9f2a1673c27af99fc11c85a336f9d77fcad7f1dd079a06581dd1229eb"} Oct 30 06:39:02.104442 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:02.104422 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" event={"ID":"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2","Type":"ContainerStarted","Data":"fa50d5a318fc55d14c815e223f6de21a2d1eef05143eec41d153193222d4397f"} Oct 30 06:39:02.105285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:02.105265 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:39:02.106497 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:02.106469 6999 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-7kz9s container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:39:02.106605 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:02.106523 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" podUID="0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:39:02.170048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:02.169984 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" podStartSLOduration=1.169963593 podStartE2EDuration="1.169963593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:39:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:39:02.1651897 +0000 UTC m=+2708.674209598" watchObservedRunningTime="2026-10-30 06:39:02.169963593 +0000 UTC m=+2708.678983481" Oct 30 06:39:02.234203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:02.231619 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" podStartSLOduration=2.182485222 podStartE2EDuration="5.231595231s" totalImagesPullingTime="3.049110009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:39:02.227366496 +0000 UTC m=+2708.736386384" watchObservedRunningTime="2026-10-30 06:39:02.231595231 +0000 UTC m=+2708.740615124" Oct 30 06:39:02.454897 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:02.448945 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:02.468895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:02.468857 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-b9r4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 06:39:02.469095 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:02.468916 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" podUID="85d15937-bbc6-440d-ba22-f7989f1e774d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 06:39:03.067362 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:03.067325 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:39:03.099083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:03.099053 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:03.116167 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:03.116129 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:39:03.457306 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:03.453771 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-b9r4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 06:39:03.457306 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:03.453852 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" podUID="85d15937-bbc6-440d-ba22-f7989f1e774d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 06:39:03.460764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:03.460731 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:04.178717 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:39:04.178673 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:39:04.452578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:04.452298 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-b9r4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 06:39:04.452578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:04.452366 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" podUID="85d15937-bbc6-440d-ba22-f7989f1e774d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 06:39:05.076439 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:05.076404 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:05.459729 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:05.459690 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-b9r4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 06:39:05.460166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:05.459760 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" podUID="85d15937-bbc6-440d-ba22-f7989f1e774d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 06:39:05.697766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:05.697737 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:06.460375 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:06.460304 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-b9r4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 06:39:06.460885 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:06.460384 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" podUID="85d15937-bbc6-440d-ba22-f7989f1e774d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 06:39:06.958243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:06.957860 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:07.452240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:07.451778 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-b9r4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 06:39:07.452240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:07.451840 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" podUID="85d15937-bbc6-440d-ba22-f7989f1e774d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 06:39:08.448791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:08.448758 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-b9r4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 06:39:08.449264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:08.448815 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" podUID="85d15937-bbc6-440d-ba22-f7989f1e774d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 06:39:08.754056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:08.753977 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:09.449731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:09.449698 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-b9r4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 06:39:09.450147 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:09.449761 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" podUID="85d15937-bbc6-440d-ba22-f7989f1e774d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 06:39:10.536619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:10.536574 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:10.587873 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:10.587831 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:10.588344 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:10.588306 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:10.590942 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:10.590917 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:39:12.330833 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:12.330801 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:14.034386 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:14.034356 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:15.762739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:15.762706 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:17.389610 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:17.389574 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:17.543057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:17.543025 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:17.834373 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:17.834284 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:19.760041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:19.759946 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:21.540423 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:21.540394 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:23.243161 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:23.243127 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:25.032142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:25.032105 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:26.797478 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:26.797444 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:28.521906 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:28.521876 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:29.428544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:29.428500 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:29.566237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:29.565609 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:30.554359 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:30.554301 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:32.359400 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:32.359367 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:34.164283 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:34.163610 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:34.219552 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:39:34.219498 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:39:36.062168 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:36.062086 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:37.754006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:37.753971 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:39.549432 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:39.549398 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:41.386110 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:41.386073 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:41.536243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:41.535563 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:42.087261 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:42.087222 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:42.231046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:42.231018 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:43.517626 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:43.517596 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:43.773812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:43.773709 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:45.030506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:45.030461 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:45.186945 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:45.186912 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:45.336137 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:45.336059 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:45.480884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:45.480837 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:45.656602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:45.656574 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:45.807971 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:45.807942 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:45.960846 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:45.960774 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:46.451764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:46.451724 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:46.593998 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:46.593927 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:47.333160 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:47.333131 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:49.098861 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:49.098828 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:49.233204 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:49.233166 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:49.378066 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:49.378035 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:51.029288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:51.029252 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:52.178866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:52.178828 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:52.330340 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:52.330310 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:52.964101 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:52.964055 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:53.452505 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:53.452468 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:53.592073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:53.592041 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:54.095179 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:54.095139 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:54.744085 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:54.744040 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:56.135276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:56.135243 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:56.291350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:56.291303 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:56.423932 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:56.423847 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:56.566065 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:56.566033 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:56.729807 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:56.729715 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:57.014016 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:57.013936 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:57.156698 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:57.156645 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:57.769176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:57.769148 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:57.915456 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:57.915426 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:58.747594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:58.747558 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:39:59.213156 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:39:59.213119 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:00.737943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:00.737906 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:01.309357 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:01.309321 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:01.462380 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:01.462348 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:01.606085 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:01.605999 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:02.604654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:02.604608 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:03.592725 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:03.592689 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:04.145323 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:40:04.145291 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:40:04.306526 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:04.306494 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:05.843318 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:05.843275 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:06.142893 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:06.142862 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:07.769629 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:07.769598 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:07.913821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:07.913786 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:08.060684 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:08.060582 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:08.251620 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:08.251582 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:08.408868 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:08.408843 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:08.571634 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:08.571584 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:08.718674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:08.718594 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:09.251058 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:09.251027 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:09.385825 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:09.385776 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:09.610891 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:09.610860 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:09.748693 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:09.748654 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:10.142703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:10.142675 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:10.959108 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:10.959081 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:11.926806 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:11.926778 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:12.222276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:12.222164 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:12.380565 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:12.380519 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:12.905949 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:12.905910 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:13.053975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:13.053940 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:13.963726 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:13.963692 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:15.809568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:15.809538 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:17.522801 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:17.522744 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:19.347688 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:19.347654 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:21.149932 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:21.149898 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:22.889280 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:22.886959 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:24.638912 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:24.638881 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:26.354097 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:26.354067 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:28.165002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:28.164964 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:28.727958 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:28.727924 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:28.894297 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:28.894262 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:29.935728 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:29.935692 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:31.682142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:31.682110 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:33.557941 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:33.557852 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:34.179276 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:40:34.179200 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:40:35.513067 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:35.513006 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:37.235021 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:37.234987 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:39.018515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:39.018484 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:39.691702 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:39.691664 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:39.833900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:39.833872 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:40.786506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:40.786473 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:42.527746 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:42.527552 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:44.419503 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:44.419450 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:46.158878 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:46.158845 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:47.933381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:47.933350 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:49.813664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:49.813627 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:51.567500 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:51.567472 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:53.512185 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:53.512143 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:55.333918 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:55.333885 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:57.145743 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:57.145711 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:58.918756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:58.918727 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:40:59.930944 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:40:59.930905 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:00.084128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:00.084039 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:00.767200 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:00.767168 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:01.049955 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:01.049871 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:01.193206 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:01.193173 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:02.626617 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:02.626579 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:03.709336 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:03.709299 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:04.144389 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:41:04.144352 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:41:04.395994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:04.395895 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:05.970057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:05.970010 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:06.301670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:06.301584 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:06.787426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:06.787394 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:08.079178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:08.079139 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:09.915701 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:09.915670 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:11.697619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:11.697591 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:13.507658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:13.507626 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:14.002176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:14.002135 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:14.136962 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:14.136855 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:14.282594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:14.282515 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:15.442607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:15.442575 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:17.427432 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:17.427405 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:19.211951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:19.211902 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:21.031440 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:21.031407 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:21.174556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:21.174522 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:22.118641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:22.118592 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:22.269342 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:22.269301 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:23.018698 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:23.018665 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:24.741835 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:24.741773 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:26.620762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:26.620729 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:27.625928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:27.625897 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:27.779904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:27.779849 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:28.307014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:28.306974 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:28.459196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:28.459150 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:28.618038 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:28.617975 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:28.768933 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:28.768416 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:29.942375 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:29.942343 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:30.091164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:30.091126 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:30.625729 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:30.625677 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:30.832915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:30.832791 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:32.601752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:32.601716 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:34.143118 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:41:34.143082 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:41:34.309779 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:34.309748 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:34.843391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:34.843364 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:34.990505 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:34.990474 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:35.140530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:35.140497 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:36.141936 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:36.141899 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:37.436447 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:37.436410 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:37.578561 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:37.578532 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:37.730852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:37.730778 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:38.073262 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:38.073148 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:38.804149 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:38.804122 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:38.976458 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:38.976416 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:40.036738 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:40.036685 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:40.184086 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:40.184042 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:40.617770 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:40.617738 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:40.752761 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:40.752729 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:41.772774 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:41.772741 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:41.913846 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:41.913813 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:42.758325 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:42.758294 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:42.904343 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:42.904307 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:43.828614 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:43.828582 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:45.399654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:45.399602 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:45.533408 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:45.533376 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:45.786536 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:45.786451 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:46.452400 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:46.452366 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:46.602799 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:46.602766 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:47.811743 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:47.811696 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:47.958918 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:47.958886 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:48.146542 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:48.146498 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:49.929430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:49.929397 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:51.712840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:51.712807 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:52.735466 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:52.735429 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:52.885256 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:52.881731 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:53.791688 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:53.791658 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:54.422999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:54.422965 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:54.582771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:54.582697 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:55.281985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:55.281952 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:55.424454 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:55.424422 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:55.990388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:55.990355 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:57.699399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:57.699354 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:57.839665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:57.839628 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:57.996141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:57.996051 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:41:59.750720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:41:59.750687 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:01.354319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.354286 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:01.504239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.494984 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:01.706575 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.706543 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf"] Oct 30 06:42:01.711493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.711473 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.739436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.739393 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf"] Oct 30 06:42:01.801053 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.801018 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.801265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.801069 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f2aec32-15b0-4a59-839b-a1e23f337319-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.801265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.801101 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.801265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.801164 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.801452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.801375 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.801452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.801420 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.801586 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.801559 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc5pn\" (UniqueName: \"kubernetes.io/projected/5f2aec32-15b0-4a59-839b-a1e23f337319-kube-api-access-nc5pn\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.801700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.801683 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.801769 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.801753 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.835448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.835417 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:01.902740 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.902702 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f2aec32-15b0-4a59-839b-a1e23f337319-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.902963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.902753 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.902963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.902768 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.902963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.902834 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.902963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.902858 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.902963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.902906 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc5pn\" (UniqueName: \"kubernetes.io/projected/5f2aec32-15b0-4a59-839b-a1e23f337319-kube-api-access-nc5pn\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.903250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.902996 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.903250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.903057 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.903250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.903173 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.903670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.903608 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.903670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.903651 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.903987 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.903966 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f2aec32-15b0-4a59-839b-a1e23f337319-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.904061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.904014 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.905778 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.905754 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.905959 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.905941 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.906359 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.906336 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.928180 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.928149 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc5pn\" (UniqueName: \"kubernetes.io/projected/5f2aec32-15b0-4a59-839b-a1e23f337319-kube-api-access-nc5pn\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:01.928391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:01.928338 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-qtzdf\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:02.026814 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:02.026734 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:02.363578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:02.363535 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf"] Oct 30 06:42:02.370023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:02.369979 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:42:02.370162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:02.370073 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:42:02.370162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:02.370121 6999 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:42:02.621033 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:02.620943 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" event={"ID":"5f2aec32-15b0-4a59-839b-a1e23f337319","Type":"ContainerStarted","Data":"e61e34e3f27e7678025586f49f31b8126dbb86cca558b26710eafefb34b0f130"} Oct 30 06:42:02.621033 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:02.621001 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" event={"ID":"5f2aec32-15b0-4a59-839b-a1e23f337319","Type":"ContainerStarted","Data":"7dbabb1f7c76a74ec86efc5dfbe49a9563f5c335fd51da29e91526ff397734c2"} Oct 30 06:42:02.678165 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:02.678106 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" podStartSLOduration=1.678088461 podStartE2EDuration="1.678088461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:42:02.669793905 +0000 UTC m=+2889.178813816" watchObservedRunningTime="2026-10-30 06:42:02.678088461 +0000 UTC m=+2889.187108350" Oct 30 06:42:02.793085 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:02.793038 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:02.949392 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:02.949311 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:03.032406 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:03.032363 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:03.032612 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:03.032537 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qtzdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:42:03.032612 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:03.032596 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" podUID="5f2aec32-15b0-4a59-839b-a1e23f337319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 06:42:03.101647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:03.101618 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:03.418448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:03.418365 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:03.813997 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:03.813916 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:03.961239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:03.957825 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:04.027595 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:04.027564 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qtzdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:42:04.027782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:04.027621 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" podUID="5f2aec32-15b0-4a59-839b-a1e23f337319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 06:42:04.098706 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:04.098325 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:04.179614 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:42:04.179256 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:42:04.248240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:04.247740 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:05.027982 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:05.027948 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qtzdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:42:05.028415 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:05.028003 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" podUID="5f2aec32-15b0-4a59-839b-a1e23f337319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 06:42:05.421041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:05.421011 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:06.028073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:06.028026 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qtzdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:42:06.028604 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:06.028090 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" podUID="5f2aec32-15b0-4a59-839b-a1e23f337319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 06:42:06.160290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:06.160127 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:06.490428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:06.490394 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:06.627384 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:06.627348 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:07.042652 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:07.042615 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qtzdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:42:07.044621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:07.044587 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" podUID="5f2aec32-15b0-4a59-839b-a1e23f337319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 06:42:08.027813 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:08.027776 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qtzdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:42:08.028002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:08.027823 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" podUID="5f2aec32-15b0-4a59-839b-a1e23f337319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 06:42:08.129909 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:08.129869 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:09.027695 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:09.027661 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qtzdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:42:09.027884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:09.027731 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" podUID="5f2aec32-15b0-4a59-839b-a1e23f337319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 06:42:09.882751 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:09.882708 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:10.027806 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:10.027767 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-qtzdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 06:42:10.027991 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:10.027837 6999 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" podUID="5f2aec32-15b0-4a59-839b-a1e23f337319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 06:42:11.066061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:11.066025 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:11.066739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:11.066714 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:11.068919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:11.068892 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:42:11.332548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:11.332459 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:11.476984 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:11.476914 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:11.846689 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:11.846656 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:13.920384 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:13.920350 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:15.837874 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:15.837831 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:17.581110 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:17.581073 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:19.324850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:19.324816 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:21.426485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:21.426422 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:22.131056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:22.131009 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:23.550734 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:23.550685 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:25.422417 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:25.422198 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:27.219113 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:27.219081 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:28.930426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:28.930389 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:30.742601 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:30.742568 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:32.525068 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:32.525035 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:34.143611 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:42:34.143574 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:42:34.302630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:34.302599 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:36.079864 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:36.079828 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:37.890874 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:37.890834 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:39.657451 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:39.657408 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:41.422863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:41.422833 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96"] Oct 30 06:42:41.423305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:41.423066 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" podUID="62421fb5-a574-4d99-a8e8-747f202e7e24" containerName="istio-proxy" containerID="cri-o://4b41a093dc882dcd94773ab6abca39dc4ef7b3149bac586881249b055d9d62be" gracePeriod=30 Oct 30 06:42:41.541766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:41.541725 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:42.160111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:42.160078 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k"] Oct 30 06:42:42.160348 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:42.160314 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" podUID="86cc6726-5087-432a-8487-5dd10dfb4c51" containerName="istio-proxy" containerID="cri-o://2baf2d096c4d9ca05d309281b63164c94225cb419e1aee7fc71b475ab1764192" gracePeriod=30 Oct 30 06:42:43.309029 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:43.308976 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:45.021681 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:45.021646 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:46.713632 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.713598 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:42:46.782252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.782192 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-workload-socket\") pod \"62421fb5-a574-4d99-a8e8-747f202e7e24\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " Oct 30 06:42:46.782425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.782288 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-podinfo\") pod \"62421fb5-a574-4d99-a8e8-747f202e7e24\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " Oct 30 06:42:46.782425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.782332 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-token\" (UniqueName: \"kubernetes.io/projected/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-token\") pod \"62421fb5-a574-4d99-a8e8-747f202e7e24\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " Oct 30 06:42:46.782542 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.782452 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-data\") pod \"62421fb5-a574-4d99-a8e8-747f202e7e24\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " Oct 30 06:42:46.782542 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.782470 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-workload-socket" pod "62421fb5-a574-4d99-a8e8-747f202e7e24" (UID: "62421fb5-a574-4d99-a8e8-747f202e7e24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:42:46.782660 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.782583 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-credential-socket\") pod \"62421fb5-a574-4d99-a8e8-747f202e7e24\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " Oct 30 06:42:46.782660 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.782642 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62421fb5-a574-4d99-a8e8-747f202e7e24-kube-api-access-jk4dw\" (UniqueName: \"kubernetes.io/projected/62421fb5-a574-4d99-a8e8-747f202e7e24-kube-api-access-jk4dw\") pod \"62421fb5-a574-4d99-a8e8-747f202e7e24\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " Oct 30 06:42:46.782772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.782678 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62421fb5-a574-4d99-a8e8-747f202e7e24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62421fb5-a574-4d99-a8e8-747f202e7e24-istiod-ca-cert\") pod \"62421fb5-a574-4d99-a8e8-747f202e7e24\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " Oct 30 06:42:46.782833 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.782765 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-data" pod "62421fb5-a574-4d99-a8e8-747f202e7e24" (UID: "62421fb5-a574-4d99-a8e8-747f202e7e24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:42:46.782833 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.782814 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-workload-certs\") pod \"62421fb5-a574-4d99-a8e8-747f202e7e24\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " Oct 30 06:42:46.783012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.782848 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-envoy\") pod \"62421fb5-a574-4d99-a8e8-747f202e7e24\" (UID: \"62421fb5-a574-4d99-a8e8-747f202e7e24\") " Oct 30 06:42:46.783012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.782875 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-credential-socket" pod "62421fb5-a574-4d99-a8e8-747f202e7e24" (UID: "62421fb5-a574-4d99-a8e8-747f202e7e24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:42:46.783289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.783228 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62421fb5-a574-4d99-a8e8-747f202e7e24-istiod-ca-cert" pod "62421fb5-a574-4d99-a8e8-747f202e7e24" (UID: "62421fb5-a574-4d99-a8e8-747f202e7e24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:42:46.783553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.783501 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-workload-certs" pod "62421fb5-a574-4d99-a8e8-747f202e7e24" (UID: "62421fb5-a574-4d99-a8e8-747f202e7e24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:42:46.784129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.784046 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:46.784129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.784074 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:46.784129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.784086 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:46.784129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.784098 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62421fb5-a574-4d99-a8e8-747f202e7e24-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:46.784129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.784109 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:46.785479 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.785441 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-token" pod "62421fb5-a574-4d99-a8e8-747f202e7e24" (UID: "62421fb5-a574-4d99-a8e8-747f202e7e24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:46.786281 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.786257 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62421fb5-a574-4d99-a8e8-747f202e7e24-kube-api-access-jk4dw" pod "62421fb5-a574-4d99-a8e8-747f202e7e24" (UID: "62421fb5-a574-4d99-a8e8-747f202e7e24"). InnerVolumeSpecName "kube-api-access-jk4dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:46.786393 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.786378 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-podinfo" pod "62421fb5-a574-4d99-a8e8-747f202e7e24" (UID: "62421fb5-a574-4d99-a8e8-747f202e7e24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:42:46.786916 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.786894 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-envoy" pod "62421fb5-a574-4d99-a8e8-747f202e7e24" (UID: "62421fb5-a574-4d99-a8e8-747f202e7e24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:42:46.887928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.887896 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk4dw\" (UniqueName: \"kubernetes.io/projected/62421fb5-a574-4d99-a8e8-747f202e7e24-kube-api-access-jk4dw\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:46.887928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.887927 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:46.887928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.887936 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:46.888207 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.887945 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62421fb5-a574-4d99-a8e8-747f202e7e24-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:46.992118 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.992032 6999 generic.go:356] "Generic (PLEG): container finished" podID="62421fb5-a574-4d99-a8e8-747f202e7e24" containerID="4b41a093dc882dcd94773ab6abca39dc4ef7b3149bac586881249b055d9d62be" exitCode=0 Oct 30 06:42:46.992118 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.992081 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" Oct 30 06:42:46.992347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.992123 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" event={"ID":"62421fb5-a574-4d99-a8e8-747f202e7e24","Type":"ContainerDied","Data":"4b41a093dc882dcd94773ab6abca39dc4ef7b3149bac586881249b055d9d62be"} Oct 30 06:42:46.992347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.992180 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96" event={"ID":"62421fb5-a574-4d99-a8e8-747f202e7e24","Type":"ContainerDied","Data":"dbfb5a29125b9e2586534d70f01deee0cfaa38a99641152f0468dffb3dcccb93"} Oct 30 06:42:46.992347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:46.992203 6999 scope.go:122] "RemoveContainer" containerID="4b41a093dc882dcd94773ab6abca39dc4ef7b3149bac586881249b055d9d62be" Oct 30 06:42:47.004015 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.004000 6999 scope.go:122] "RemoveContainer" containerID="4b41a093dc882dcd94773ab6abca39dc4ef7b3149bac586881249b055d9d62be" Oct 30 06:42:47.004352 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:42:47.004333 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b41a093dc882dcd94773ab6abca39dc4ef7b3149bac586881249b055d9d62be\": container with ID starting with 4b41a093dc882dcd94773ab6abca39dc4ef7b3149bac586881249b055d9d62be not found: ID does not exist" containerID="4b41a093dc882dcd94773ab6abca39dc4ef7b3149bac586881249b055d9d62be" Oct 30 06:42:47.004422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.004363 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b41a093dc882dcd94773ab6abca39dc4ef7b3149bac586881249b055d9d62be"} err="failed to get container status \"4b41a093dc882dcd94773ab6abca39dc4ef7b3149bac586881249b055d9d62be\": rpc error: code = NotFound desc = could not find container \"4b41a093dc882dcd94773ab6abca39dc4ef7b3149bac586881249b055d9d62be\": container with ID starting with 4b41a093dc882dcd94773ab6abca39dc4ef7b3149bac586881249b055d9d62be not found: ID does not exist" Oct 30 06:42:47.020723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.020706 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:47.070431 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.070398 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96"] Oct 30 06:42:47.074006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.073976 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6nb96"] Oct 30 06:42:47.429350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.429324 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:42:47.495391 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.495359 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-workload-certs\") pod \"86cc6726-5087-432a-8487-5dd10dfb4c51\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " Oct 30 06:42:47.495556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.495426 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-data\") pod \"86cc6726-5087-432a-8487-5dd10dfb4c51\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " Oct 30 06:42:47.495556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.495466 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-envoy\") pod \"86cc6726-5087-432a-8487-5dd10dfb4c51\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " Oct 30 06:42:47.495556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.495511 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-token\" (UniqueName: \"kubernetes.io/projected/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-token\") pod \"86cc6726-5087-432a-8487-5dd10dfb4c51\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " Oct 30 06:42:47.495789 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.495628 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86cc6726-5087-432a-8487-5dd10dfb4c51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86cc6726-5087-432a-8487-5dd10dfb4c51-istiod-ca-cert\") pod \"86cc6726-5087-432a-8487-5dd10dfb4c51\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " Oct 30 06:42:47.495789 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.495694 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-workload-certs" pod "86cc6726-5087-432a-8487-5dd10dfb4c51" (UID: "86cc6726-5087-432a-8487-5dd10dfb4c51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:42:47.495789 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.495759 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86cc6726-5087-432a-8487-5dd10dfb4c51-kube-api-access-snpvf\" (UniqueName: \"kubernetes.io/projected/86cc6726-5087-432a-8487-5dd10dfb4c51-kube-api-access-snpvf\") pod \"86cc6726-5087-432a-8487-5dd10dfb4c51\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " Oct 30 06:42:47.496114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.495842 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-podinfo\") pod \"86cc6726-5087-432a-8487-5dd10dfb4c51\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " Oct 30 06:42:47.496114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.495901 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-credential-socket\") pod \"86cc6726-5087-432a-8487-5dd10dfb4c51\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " Oct 30 06:42:47.496114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.495939 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-workload-socket\") pod \"86cc6726-5087-432a-8487-5dd10dfb4c51\" (UID: \"86cc6726-5087-432a-8487-5dd10dfb4c51\") " Oct 30 06:42:47.496114 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.495759 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-data" pod "86cc6726-5087-432a-8487-5dd10dfb4c51" (UID: "86cc6726-5087-432a-8487-5dd10dfb4c51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:42:47.496436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.496098 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86cc6726-5087-432a-8487-5dd10dfb4c51-istiod-ca-cert" pod "86cc6726-5087-432a-8487-5dd10dfb4c51" (UID: "86cc6726-5087-432a-8487-5dd10dfb4c51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:42:47.496816 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.496794 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:47.496926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.496824 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:47.496926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.496838 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86cc6726-5087-432a-8487-5dd10dfb4c51-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:47.497062 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.496950 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-workload-socket" pod "86cc6726-5087-432a-8487-5dd10dfb4c51" (UID: "86cc6726-5087-432a-8487-5dd10dfb4c51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:42:47.497273 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.497198 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-credential-socket" pod "86cc6726-5087-432a-8487-5dd10dfb4c51" (UID: "86cc6726-5087-432a-8487-5dd10dfb4c51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:42:47.500034 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.499987 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86cc6726-5087-432a-8487-5dd10dfb4c51-kube-api-access-snpvf" pod "86cc6726-5087-432a-8487-5dd10dfb4c51" (UID: "86cc6726-5087-432a-8487-5dd10dfb4c51"). InnerVolumeSpecName "kube-api-access-snpvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:47.500490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.500462 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-podinfo" pod "86cc6726-5087-432a-8487-5dd10dfb4c51" (UID: "86cc6726-5087-432a-8487-5dd10dfb4c51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:42:47.500599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.500557 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-token" pod "86cc6726-5087-432a-8487-5dd10dfb4c51" (UID: "86cc6726-5087-432a-8487-5dd10dfb4c51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:42:47.500599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.500576 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-envoy" pod "86cc6726-5087-432a-8487-5dd10dfb4c51" (UID: "86cc6726-5087-432a-8487-5dd10dfb4c51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:42:47.598371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.598298 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snpvf\" (UniqueName: \"kubernetes.io/projected/86cc6726-5087-432a-8487-5dd10dfb4c51-kube-api-access-snpvf\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:47.598371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.598324 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:47.598371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.598334 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:47.598371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.598343 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:47.598371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.598351 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:47.598371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.598371 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86cc6726-5087-432a-8487-5dd10dfb4c51-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:42:47.612197 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:47.612164 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62421fb5-a574-4d99-a8e8-747f202e7e24" path="/var/lib/kubelet/pods/62421fb5-a574-4d99-a8e8-747f202e7e24/volumes" Oct 30 06:42:48.000075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:48.000045 6999 generic.go:356] "Generic (PLEG): container finished" podID="86cc6726-5087-432a-8487-5dd10dfb4c51" containerID="2baf2d096c4d9ca05d309281b63164c94225cb419e1aee7fc71b475ab1764192" exitCode=0 Oct 30 06:42:48.000554 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:48.000092 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" Oct 30 06:42:48.000554 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:48.000130 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" event={"ID":"86cc6726-5087-432a-8487-5dd10dfb4c51","Type":"ContainerDied","Data":"2baf2d096c4d9ca05d309281b63164c94225cb419e1aee7fc71b475ab1764192"} Oct 30 06:42:48.000554 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:48.000183 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k" event={"ID":"86cc6726-5087-432a-8487-5dd10dfb4c51","Type":"ContainerDied","Data":"c44d90bd130b42ce558fc251e1a43a792df4bd0a4d13868d7adf612e9c2b5260"} Oct 30 06:42:48.000554 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:48.000243 6999 scope.go:122] "RemoveContainer" containerID="2baf2d096c4d9ca05d309281b63164c94225cb419e1aee7fc71b475ab1764192" Oct 30 06:42:48.011779 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:48.011758 6999 scope.go:122] "RemoveContainer" containerID="2baf2d096c4d9ca05d309281b63164c94225cb419e1aee7fc71b475ab1764192" Oct 30 06:42:48.012096 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:42:48.012068 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2baf2d096c4d9ca05d309281b63164c94225cb419e1aee7fc71b475ab1764192\": container with ID starting with 2baf2d096c4d9ca05d309281b63164c94225cb419e1aee7fc71b475ab1764192 not found: ID does not exist" containerID="2baf2d096c4d9ca05d309281b63164c94225cb419e1aee7fc71b475ab1764192" Oct 30 06:42:48.012178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:48.012102 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2baf2d096c4d9ca05d309281b63164c94225cb419e1aee7fc71b475ab1764192"} err="failed to get container status \"2baf2d096c4d9ca05d309281b63164c94225cb419e1aee7fc71b475ab1764192\": rpc error: code = NotFound desc = could not find container \"2baf2d096c4d9ca05d309281b63164c94225cb419e1aee7fc71b475ab1764192\": container with ID starting with 2baf2d096c4d9ca05d309281b63164c94225cb419e1aee7fc71b475ab1764192 not found: ID does not exist" Oct 30 06:42:48.053067 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:48.053042 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k"] Oct 30 06:42:48.057999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:48.057977 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cfg5k"] Oct 30 06:42:48.815829 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:48.815803 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:49.611925 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:49.611893 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86cc6726-5087-432a-8487-5dd10dfb4c51" path="/var/lib/kubelet/pods/86cc6726-5087-432a-8487-5dd10dfb4c51/volumes" Oct 30 06:42:50.552167 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:50.552134 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:52.443466 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:52.443435 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:54.234822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:54.234787 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:55.950571 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:55.950540 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:57.730658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:57.730626 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:42:59.521635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:42:59.521606 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:01.316238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:01.312599 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:03.493883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:03.493817 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:04.096788 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:04.096753 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:04.216847 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:43:04.216802 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:43:04.238099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:04.238065 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:05.519810 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:05.519768 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:06.639280 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:06.639244 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:07.282473 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:07.282435 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:09.055798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:09.055757 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:10.750188 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:10.750151 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:12.596148 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:12.596113 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:14.344393 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:14.344362 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:16.135916 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:16.135878 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:17.937544 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:17.937511 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:19.712622 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:19.712562 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:20.571886 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:20.571852 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:21.435073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:21.435040 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:23.220471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:23.220439 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:25.041089 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:25.041048 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:26.741906 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:26.741870 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:28.533651 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:28.533623 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:30.261364 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:30.261332 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:32.037746 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:32.037708 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:33.822984 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:33.822755 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:34.143376 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:43:34.143338 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:43:35.658129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:35.658091 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:37.446377 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:37.446346 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:39.259474 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:39.259437 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:41.009458 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:41.009426 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:42.753937 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:42.753894 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:44.520563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:44.520531 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:46.335350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:46.335314 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:48.068948 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:48.068914 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:49.939336 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:49.939304 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:51.731826 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:51.731789 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:53.436609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:53.436567 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:54.949106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:54.949015 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6"] Oct 30 06:43:54.949566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:54.949263 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" podUID="b253e877-6873-4e91-a50b-30fef43bcd3e" containerName="istio-proxy" containerID="cri-o://e1218689aa44472520947f883c287bc6a668ade75f5f6582cb6963408d8555a1" gracePeriod=30 Oct 30 06:43:55.255077 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:55.254990 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:57.127765 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:57.127729 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:43:58.278569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:58.278530 6999 patch_prober.go:29] interesting pod/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:43:58.278960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:58.278638 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" podUID="b253e877-6873-4e91-a50b-30fef43bcd3e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:43:58.920603 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:43:58.920565 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:00.255453 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.255419 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:44:00.399504 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.399470 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-podinfo\") pod \"b253e877-6873-4e91-a50b-30fef43bcd3e\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " Oct 30 06:44:00.399680 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.399563 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-token\" (UniqueName: \"kubernetes.io/projected/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-token\") pod \"b253e877-6873-4e91-a50b-30fef43bcd3e\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " Oct 30 06:44:00.399680 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.399606 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-credential-socket\") pod \"b253e877-6873-4e91-a50b-30fef43bcd3e\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " Oct 30 06:44:00.399680 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.399642 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-envoy\") pod \"b253e877-6873-4e91-a50b-30fef43bcd3e\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " Oct 30 06:44:00.399841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.399683 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b253e877-6873-4e91-a50b-30fef43bcd3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b253e877-6873-4e91-a50b-30fef43bcd3e-istiod-ca-cert\") pod \"b253e877-6873-4e91-a50b-30fef43bcd3e\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " Oct 30 06:44:00.399841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.399719 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-workload-certs\") pod \"b253e877-6873-4e91-a50b-30fef43bcd3e\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " Oct 30 06:44:00.399841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.399764 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-workload-socket\") pod \"b253e877-6873-4e91-a50b-30fef43bcd3e\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " Oct 30 06:44:00.400014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.399908 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-credential-socket" pod "b253e877-6873-4e91-a50b-30fef43bcd3e" (UID: "b253e877-6873-4e91-a50b-30fef43bcd3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:00.400082 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.400044 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b253e877-6873-4e91-a50b-30fef43bcd3e-kube-api-access-dkmhz\" (UniqueName: \"kubernetes.io/projected/b253e877-6873-4e91-a50b-30fef43bcd3e-kube-api-access-dkmhz\") pod \"b253e877-6873-4e91-a50b-30fef43bcd3e\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " Oct 30 06:44:00.400082 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.400047 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-workload-socket" pod "b253e877-6873-4e91-a50b-30fef43bcd3e" (UID: "b253e877-6873-4e91-a50b-30fef43bcd3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:00.400197 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.400090 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-data\") pod \"b253e877-6873-4e91-a50b-30fef43bcd3e\" (UID: \"b253e877-6873-4e91-a50b-30fef43bcd3e\") " Oct 30 06:44:00.400197 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.400141 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b253e877-6873-4e91-a50b-30fef43bcd3e-istiod-ca-cert" pod "b253e877-6873-4e91-a50b-30fef43bcd3e" (UID: "b253e877-6873-4e91-a50b-30fef43bcd3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:44:00.400367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.400280 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-workload-certs" pod "b253e877-6873-4e91-a50b-30fef43bcd3e" (UID: "b253e877-6873-4e91-a50b-30fef43bcd3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:00.400577 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.400506 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-data" pod "b253e877-6873-4e91-a50b-30fef43bcd3e" (UID: "b253e877-6873-4e91-a50b-30fef43bcd3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:00.400991 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.400969 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.401087 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.400996 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.401087 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.401011 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.401087 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.401026 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b253e877-6873-4e91-a50b-30fef43bcd3e-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.401087 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.401040 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.401923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.401898 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-podinfo" pod "b253e877-6873-4e91-a50b-30fef43bcd3e" (UID: "b253e877-6873-4e91-a50b-30fef43bcd3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:44:00.402088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.402057 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-token" pod "b253e877-6873-4e91-a50b-30fef43bcd3e" (UID: "b253e877-6873-4e91-a50b-30fef43bcd3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:44:00.402444 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.402423 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b253e877-6873-4e91-a50b-30fef43bcd3e-kube-api-access-dkmhz" pod "b253e877-6873-4e91-a50b-30fef43bcd3e" (UID: "b253e877-6873-4e91-a50b-30fef43bcd3e"). InnerVolumeSpecName "kube-api-access-dkmhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:44:00.402571 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.402519 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-envoy" pod "b253e877-6873-4e91-a50b-30fef43bcd3e" (UID: "b253e877-6873-4e91-a50b-30fef43bcd3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:44:00.502837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.502759 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkmhz\" (UniqueName: \"kubernetes.io/projected/b253e877-6873-4e91-a50b-30fef43bcd3e-kube-api-access-dkmhz\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.502837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.502789 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.502837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.502799 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.502837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.502808 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b253e877-6873-4e91-a50b-30fef43bcd3e-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:44:00.558385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.558351 6999 generic.go:356] "Generic (PLEG): container finished" podID="b253e877-6873-4e91-a50b-30fef43bcd3e" containerID="e1218689aa44472520947f883c287bc6a668ade75f5f6582cb6963408d8555a1" exitCode=0 Oct 30 06:44:00.558568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.558404 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" Oct 30 06:44:00.558568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.558434 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" event={"ID":"b253e877-6873-4e91-a50b-30fef43bcd3e","Type":"ContainerDied","Data":"e1218689aa44472520947f883c287bc6a668ade75f5f6582cb6963408d8555a1"} Oct 30 06:44:00.558568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.558486 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6" event={"ID":"b253e877-6873-4e91-a50b-30fef43bcd3e","Type":"ContainerDied","Data":"62d3228869a96ce500f57131a93759486321eb3390dc169a5db7fc90ab658746"} Oct 30 06:44:00.558568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.558509 6999 scope.go:122] "RemoveContainer" containerID="e1218689aa44472520947f883c287bc6a668ade75f5f6582cb6963408d8555a1" Oct 30 06:44:00.569815 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.569798 6999 scope.go:122] "RemoveContainer" containerID="e1218689aa44472520947f883c287bc6a668ade75f5f6582cb6963408d8555a1" Oct 30 06:44:00.570109 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:44:00.570089 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1218689aa44472520947f883c287bc6a668ade75f5f6582cb6963408d8555a1\": container with ID starting with e1218689aa44472520947f883c287bc6a668ade75f5f6582cb6963408d8555a1 not found: ID does not exist" containerID="e1218689aa44472520947f883c287bc6a668ade75f5f6582cb6963408d8555a1" Oct 30 06:44:00.570197 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.570120 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1218689aa44472520947f883c287bc6a668ade75f5f6582cb6963408d8555a1"} err="failed to get container status \"e1218689aa44472520947f883c287bc6a668ade75f5f6582cb6963408d8555a1\": rpc error: code = NotFound desc = could not find container \"e1218689aa44472520947f883c287bc6a668ade75f5f6582cb6963408d8555a1\": container with ID starting with e1218689aa44472520947f883c287bc6a668ade75f5f6582cb6963408d8555a1 not found: ID does not exist" Oct 30 06:44:00.614218 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.614191 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6"] Oct 30 06:44:00.626485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.626465 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-f8nz6"] Oct 30 06:44:00.747771 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:00.747744 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:01.609812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:01.609780 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b253e877-6873-4e91-a50b-30fef43bcd3e" path="/var/lib/kubelet/pods/b253e877-6873-4e91-a50b-30fef43bcd3e/volumes" Oct 30 06:44:02.544560 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:02.544519 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:04.145068 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:44:04.145029 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:44:04.222958 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:04.222926 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:04.372823 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:04.372793 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:05.968285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:05.968207 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:06.764723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:06.764696 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:07.739913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:07.739884 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:09.521339 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:09.520998 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:11.301348 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:11.301307 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:13.142318 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:13.142290 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:14.927491 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:14.927460 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:16.641854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:16.641819 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:18.464599 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:18.464563 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:19.423116 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:19.423079 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:19.561663 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:19.561629 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:20.650991 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:20.650954 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:21.276737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:21.276706 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:21.425290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:21.425256 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:21.567920 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:21.567836 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:21.905008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:21.904974 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:22.072260 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:22.072195 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:22.222022 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:22.221936 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:22.531282 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:22.531186 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:24.425458 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:24.425422 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:26.201491 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:26.201442 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:27.916490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:27.916457 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:28.398701 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:28.398661 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:28.562269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:28.556773 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:30.082081 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:30.082047 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:30.713127 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:30.713092 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:31.756267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:31.756233 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:32.019766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:32.019683 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:33.824603 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:33.824571 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:34.143178 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:44:34.143135 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:44:35.593282 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:35.593249 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:37.348410 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:37.348374 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:39.327682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:39.327624 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:41.574265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:41.563759 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:41.642403 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:41.642354 6999 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-b9r4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:44:41.642742 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:41.642690 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" podUID="85d15937-bbc6-440d-ba22-f7989f1e774d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:44:43.324882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:43.324848 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:45.061405 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:45.061359 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:46.839997 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:46.839965 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:47.485033 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:47.484998 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:47.644516 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:47.644484 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:48.733078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:48.733015 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:50.427700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:50.427660 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:52.250516 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:52.250479 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:54.043231 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:54.043183 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:55.793264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:55.793206 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:57.537897 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:57.537860 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:44:59.349273 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:44:59.349234 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:01.248796 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:01.248766 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:03.131812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:03.131776 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:04.145550 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:45:04.144525 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:45:04.390977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:04.390909 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:05.046330 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:05.046298 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:05.667133 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:05.667092 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:05.814060 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:05.814008 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:06.896678 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:06.896645 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:07.069700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:07.069638 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:07.339278 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:07.339126 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:07.505120 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:07.505056 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:07.708694 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:07.708662 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:07.990915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:07.989780 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:08.685355 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:08.685320 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:08.947988 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:08.947897 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:10.846851 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:10.846811 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:12.614725 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:12.614692 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:14.374013 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:14.373976 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:15.132509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:15.132480 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:15.272986 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:15.272955 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:15.645137 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:15.645104 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:16.133562 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:16.133534 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:16.293916 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:16.293869 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:18.153271 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:18.153233 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:20.130392 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:20.130357 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:20.281198 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:20.281152 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:20.440581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:20.440494 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:21.738844 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:21.738815 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:21.888063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:21.888033 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:22.043061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:22.042975 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:22.192975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:22.192941 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:23.979247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:23.979192 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:25.776458 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:25.776420 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:27.648798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:27.648747 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:29.189099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:29.189068 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:29.333618 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:29.333592 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:29.480563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:29.480462 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:31.208383 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:31.208353 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:33.066881 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:33.066085 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:34.142295 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:45:34.142255 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:45:34.845373 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:34.845338 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:36.630278 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:36.630247 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:38.446432 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:38.446337 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:40.124858 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:40.124824 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:41.932166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:41.932128 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:43.740279 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:43.740237 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:45.608555 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:45.608513 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:47.321500 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:47.321466 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:49.054620 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:49.054581 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:50.855797 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:50.855758 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:52.641276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:52.641177 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:54.429092 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:54.429038 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:56.215918 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:56.215883 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:57.957195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:57.954842 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:45:59.726153 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:45:59.726111 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:01.435330 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:01.435295 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:03.209777 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:03.209743 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:03.356906 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:03.356863 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:04.150564 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:46:04.150498 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:46:04.715231 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:04.715189 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:05.137604 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:05.137559 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:06.998976 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:06.998940 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:07.163230 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:07.163181 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:08.848708 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:08.848666 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:10.624621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:10.624586 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:12.404048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:12.404014 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:14.124121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:14.124088 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:15.924431 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:15.924387 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:17.751299 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:17.751262 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:19.595790 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:19.595747 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:21.353412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:21.353375 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:23.146804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:23.146294 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:24.857923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:24.857890 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:26.540731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:26.540695 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:26.700367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:26.700329 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:28.526483 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:28.526451 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:30.226827 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:30.226793 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:31.940709 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:31.940680 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:33.729203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:33.729100 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:34.141683 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:46:34.141643 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:46:35.504812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:35.504779 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:37.354734 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:37.354692 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:39.242972 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:39.242940 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:41.044042 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:41.043994 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:42.821566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:42.821530 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:44.658081 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:44.658043 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:46.441651 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:46.441600 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:48.131355 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:48.131314 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:49.984959 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:49.984921 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:51.913306 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:51.913273 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:52.561228 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:52.561186 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:53.050969 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:53.050932 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:53.572510 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:53.572474 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:53.801507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:53.801455 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:53.962894 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:53.962858 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:55.662259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:55.662194 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:57.630442 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:57.630415 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:57.787436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:57.787407 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:57.931824 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:57.931735 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:59.403694 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:59.403659 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:46:59.544924 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:46:59.544881 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:01.430751 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:01.430720 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:03.175848 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:03.167231 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:04.145755 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:47:04.145713 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:47:04.307910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:04.307876 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:04.462258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:04.462152 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:04.898627 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:04.898597 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:05.109087 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:05.109052 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:06.711436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:06.711399 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:06.853891 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:06.853865 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:07.109649 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:07.109608 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:07.490753 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:07.490664 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:08.897596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:08.897555 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:10.599530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:10.599497 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:12.368122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:12.368094 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:13.195129 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:13.195090 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:13.353356 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:13.353321 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:14.432549 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:14.431590 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:15.654707 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:15.654670 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:15.801582 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:15.801542 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:16.138069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:16.138042 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:16.320594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:16.320564 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:16.791752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:16.791717 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:17.303338 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:17.303301 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:18.122700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:18.122670 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:20.011709 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:20.011648 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:21.223428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:21.223396 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:21.459264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:21.459235 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:22.045933 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:22.045891 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:22.963205 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:22.963166 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:23.131779 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:23.131742 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:23.953047 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:23.953009 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:25.775915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:25.775865 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:27.206875 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:27.206836 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r"] Oct 30 06:47:27.207394 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:27.207024 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" podUID="c3e004f9-f184-4b09-835a-b314650513d4" containerName="istio-proxy" containerID="cri-o://dcd27fccf95c79d87b5b36d4046ae1f19e8293551800b155b0c77fc718e6cc5f" gracePeriod=30 Oct 30 06:47:27.219250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:27.219206 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9"] Oct 30 06:47:27.219481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:27.219443 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" podUID="c3934fa5-c92d-4efd-b675-e9764ab0fcfa" containerName="istio-proxy" containerID="cri-o://714ae399d71b09d70ec1fcf5f637b14425595ff8f0621c56baed7f9e7ba64d20" gracePeriod=30 Oct 30 06:47:27.528899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:27.528810 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr"] Oct 30 06:47:27.529075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:27.529052 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" podUID="be652fcd-c5e8-4086-a10a-f80ded1187f4" containerName="istio-proxy" containerID="cri-o://50e45b5fcb9fcdb880384d14609ab3560e3f623a88f16aeb7f7dcb47b6deb1a1" gracePeriod=30 Oct 30 06:47:27.532305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:27.532277 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj"] Oct 30 06:47:27.532541 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:27.532498 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" podUID="0bc0b89a-0fb5-4632-b6e2-d8586cd93435" containerName="istio-proxy" containerID="cri-o://c520d227fef6f43bb0c39ff7df673d827acfab5fe3ad7bb225fac714cf2812cf" gracePeriod=30 Oct 30 06:47:27.535911 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:27.535887 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4"] Oct 30 06:47:27.536115 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:27.536095 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" podUID="567e2929-da18-45e4-a1e4-4e86f1ad1855" containerName="istio-proxy" containerID="cri-o://8d8b4768453bc5d1391d988e6b836c8fa81e9d69885ed9f432da054567e0fa66" gracePeriod=30 Oct 30 06:47:27.651536 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:27.651504 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:28.820821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:28.820771 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s"] Oct 30 06:47:28.821274 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:28.821019 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" podUID="0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2" containerName="discovery" containerID="cri-o://fa50d5a318fc55d14c815e223f6de21a2d1eef05143eec41d153193222d4397f" gracePeriod=30 Oct 30 06:47:29.122406 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.122380 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:47:29.243444 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.243412 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-kube-api-access-bm9bc\" (UniqueName: \"kubernetes.io/projected/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-kube-api-access-bm9bc\") pod \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " Oct 30 06:47:29.243637 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.243551 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-csr-dns-cert\") pod \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " Oct 30 06:47:29.243637 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.243586 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-kubeconfig\") pod \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " Oct 30 06:47:29.243637 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.243633 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-cacerts\" (UniqueName: \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-cacerts\") pod \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " Oct 30 06:47:29.243855 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.243712 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-local-certs\") pod \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " Oct 30 06:47:29.243855 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.243790 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-csr-ca-configmap\") pod \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " Oct 30 06:47:29.243974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.243852 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-token\" (UniqueName: \"kubernetes.io/projected/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-token\") pod \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\" (UID: \"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2\") " Oct 30 06:47:29.244433 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.244401 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-csr-ca-configmap" pod "0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2" (UID: "0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:29.244730 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.244682 6999 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-csr-ca-configmap\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:29.246782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.246501 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-local-certs" pod "0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2" (UID: "0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:29.246782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.246538 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-csr-dns-cert" pod "0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2" (UID: "0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:47:29.246958 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.246844 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-kube-api-access-bm9bc" pod "0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2" (UID: "0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2"). InnerVolumeSpecName "kube-api-access-bm9bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:29.247047 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.247026 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-kubeconfig" pod "0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2" (UID: "0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:47:29.247137 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.247104 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-cacerts" pod "0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2" (UID: "0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:47:29.248276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.248227 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-token" pod "0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2" (UID: "0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:29.345792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.345710 6999 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-local-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:29.345792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.345737 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:29.345792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.345751 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm9bc\" (UniqueName: \"kubernetes.io/projected/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-kube-api-access-bm9bc\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:29.345792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.345763 6999 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-csr-dns-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:29.345792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.345776 6999 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-istio-kubeconfig\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:29.345792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.345788 6999 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2-cacerts\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:29.390034 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.390004 6999 generic.go:356] "Generic (PLEG): container finished" podID="0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2" containerID="fa50d5a318fc55d14c815e223f6de21a2d1eef05143eec41d153193222d4397f" exitCode=0 Oct 30 06:47:29.390191 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.390048 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" Oct 30 06:47:29.390191 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.390089 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" event={"ID":"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2","Type":"ContainerDied","Data":"fa50d5a318fc55d14c815e223f6de21a2d1eef05143eec41d153193222d4397f"} Oct 30 06:47:29.390191 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.390139 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s" event={"ID":"0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2","Type":"ContainerDied","Data":"38c4cd712d7580894e724da6b032877a17ff3fa1ceb77c336429201355b452da"} Oct 30 06:47:29.390191 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.390161 6999 scope.go:122] "RemoveContainer" containerID="fa50d5a318fc55d14c815e223f6de21a2d1eef05143eec41d153193222d4397f" Oct 30 06:47:29.402088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.402066 6999 scope.go:122] "RemoveContainer" containerID="fa50d5a318fc55d14c815e223f6de21a2d1eef05143eec41d153193222d4397f" Oct 30 06:47:29.402373 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:47:29.402354 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa50d5a318fc55d14c815e223f6de21a2d1eef05143eec41d153193222d4397f\": container with ID starting with fa50d5a318fc55d14c815e223f6de21a2d1eef05143eec41d153193222d4397f not found: ID does not exist" containerID="fa50d5a318fc55d14c815e223f6de21a2d1eef05143eec41d153193222d4397f" Oct 30 06:47:29.402448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.402386 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa50d5a318fc55d14c815e223f6de21a2d1eef05143eec41d153193222d4397f"} err="failed to get container status \"fa50d5a318fc55d14c815e223f6de21a2d1eef05143eec41d153193222d4397f\": rpc error: code = NotFound desc = could not find container \"fa50d5a318fc55d14c815e223f6de21a2d1eef05143eec41d153193222d4397f\": container with ID starting with fa50d5a318fc55d14c815e223f6de21a2d1eef05143eec41d153193222d4397f not found: ID does not exist" Oct 30 06:47:29.444548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.444524 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s"] Oct 30 06:47:29.456416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.456392 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7kz9s"] Oct 30 06:47:29.533001 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.532981 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:29.612183 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:29.612154 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2" path="/var/lib/kubelet/pods/0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2/volumes" Oct 30 06:47:31.309419 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:31.309381 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:32.424265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.420635 6999 generic.go:356] "Generic (PLEG): container finished" podID="c3e004f9-f184-4b09-835a-b314650513d4" containerID="dcd27fccf95c79d87b5b36d4046ae1f19e8293551800b155b0c77fc718e6cc5f" exitCode=0 Oct 30 06:47:32.424265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.420782 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" event={"ID":"c3e004f9-f184-4b09-835a-b314650513d4","Type":"ContainerDied","Data":"dcd27fccf95c79d87b5b36d4046ae1f19e8293551800b155b0c77fc718e6cc5f"} Oct 30 06:47:32.424265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.422572 6999 generic.go:356] "Generic (PLEG): container finished" podID="c3934fa5-c92d-4efd-b675-e9764ab0fcfa" containerID="714ae399d71b09d70ec1fcf5f637b14425595ff8f0621c56baed7f9e7ba64d20" exitCode=0 Oct 30 06:47:32.424265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.422630 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" event={"ID":"c3934fa5-c92d-4efd-b675-e9764ab0fcfa","Type":"ContainerDied","Data":"714ae399d71b09d70ec1fcf5f637b14425595ff8f0621c56baed7f9e7ba64d20"} Oct 30 06:47:32.513310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.513286 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:47:32.582298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.582270 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 06:47:32.582421 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.582338 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" podUID="0bc0b89a-0fb5-4632-b6e2-d8586cd93435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 06:47:32.693910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.693878 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-workload-socket\") pod \"c3e004f9-f184-4b09-835a-b314650513d4\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " Oct 30 06:47:32.694106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.693918 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-credential-socket\") pod \"c3e004f9-f184-4b09-835a-b314650513d4\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " Oct 30 06:47:32.694106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.693948 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-istio-envoy\") pod \"c3e004f9-f184-4b09-835a-b314650513d4\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " Oct 30 06:47:32.694106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.693985 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3e004f9-f184-4b09-835a-b314650513d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e004f9-f184-4b09-835a-b314650513d4-istio-podinfo\") pod \"c3e004f9-f184-4b09-835a-b314650513d4\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " Oct 30 06:47:32.694106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.694066 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-workload-certs\") pod \"c3e004f9-f184-4b09-835a-b314650513d4\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " Oct 30 06:47:32.694387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.694169 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3e004f9-f184-4b09-835a-b314650513d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e004f9-f184-4b09-835a-b314650513d4-istiod-ca-cert\") pod \"c3e004f9-f184-4b09-835a-b314650513d4\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " Oct 30 06:47:32.694387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.694270 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-istio-data\") pod \"c3e004f9-f184-4b09-835a-b314650513d4\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " Oct 30 06:47:32.694387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.694360 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e004f9-f184-4b09-835a-b314650513d4-istio-token\" (UniqueName: \"kubernetes.io/projected/c3e004f9-f184-4b09-835a-b314650513d4-istio-token\") pod \"c3e004f9-f184-4b09-835a-b314650513d4\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " Oct 30 06:47:32.694546 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.694437 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e004f9-f184-4b09-835a-b314650513d4-kube-api-access-b9cw8\" (UniqueName: \"kubernetes.io/projected/c3e004f9-f184-4b09-835a-b314650513d4-kube-api-access-b9cw8\") pod \"c3e004f9-f184-4b09-835a-b314650513d4\" (UID: \"c3e004f9-f184-4b09-835a-b314650513d4\") " Oct 30 06:47:32.696290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.696145 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-workload-socket" pod "c3e004f9-f184-4b09-835a-b314650513d4" (UID: "c3e004f9-f184-4b09-835a-b314650513d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.696290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.696185 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-workload-certs" pod "c3e004f9-f184-4b09-835a-b314650513d4" (UID: "c3e004f9-f184-4b09-835a-b314650513d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.696655 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.696634 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-credential-socket" pod "c3e004f9-f184-4b09-835a-b314650513d4" (UID: "c3e004f9-f184-4b09-835a-b314650513d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.696765 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.696744 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-istio-data" pod "c3e004f9-f184-4b09-835a-b314650513d4" (UID: "c3e004f9-f184-4b09-835a-b314650513d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.697065 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.697045 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e004f9-f184-4b09-835a-b314650513d4-istiod-ca-cert" pod "c3e004f9-f184-4b09-835a-b314650513d4" (UID: "c3e004f9-f184-4b09-835a-b314650513d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:32.698293 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.698257 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body= Oct 30 06:47:32.698428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.698321 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" podUID="be652fcd-c5e8-4086-a10a-f80ded1187f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" Oct 30 06:47:32.699030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.699005 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e004f9-f184-4b09-835a-b314650513d4-istio-token" pod "c3e004f9-f184-4b09-835a-b314650513d4" (UID: "c3e004f9-f184-4b09-835a-b314650513d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:32.700992 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.700969 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e004f9-f184-4b09-835a-b314650513d4-kube-api-access-b9cw8" pod "c3e004f9-f184-4b09-835a-b314650513d4" (UID: "c3e004f9-f184-4b09-835a-b314650513d4"). InnerVolumeSpecName "kube-api-access-b9cw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:32.701590 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.701507 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-istio-envoy" pod "c3e004f9-f184-4b09-835a-b314650513d4" (UID: "c3e004f9-f184-4b09-835a-b314650513d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.701590 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.701552 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3e004f9-f184-4b09-835a-b314650513d4-istio-podinfo" pod "c3e004f9-f184-4b09-835a-b314650513d4" (UID: "c3e004f9-f184-4b09-835a-b314650513d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:32.732030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.731744 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:47:32.796614 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.796578 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.796614 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.796616 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.796917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.796630 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.796917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.796646 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e004f9-f184-4b09-835a-b314650513d4-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.796917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.796661 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.796917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.796674 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e004f9-f184-4b09-835a-b314650513d4-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.796917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.796686 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e004f9-f184-4b09-835a-b314650513d4-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.796917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.796698 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e004f9-f184-4b09-835a-b314650513d4-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.796917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.796710 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9cw8\" (UniqueName: \"kubernetes.io/projected/c3e004f9-f184-4b09-835a-b314650513d4-kube-api-access-b9cw8\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.838155 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.838125 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:47:32.884347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.884314 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:47:32.905321 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905287 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/567e2929-da18-45e4-a1e4-4e86f1ad1855-kube-api-access-n79gb\" (UniqueName: \"kubernetes.io/projected/567e2929-da18-45e4-a1e4-4e86f1ad1855-kube-api-access-n79gb\") pod \"567e2929-da18-45e4-a1e4-4e86f1ad1855\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " Oct 30 06:47:32.905576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905367 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-podinfo\") pod \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " Oct 30 06:47:32.905576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905410 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-envoy\") pod \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " Oct 30 06:47:32.905576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905438 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be652fcd-c5e8-4086-a10a-f80ded1187f4-kube-api-access-97289\" (UniqueName: \"kubernetes.io/projected/be652fcd-c5e8-4086-a10a-f80ded1187f4-kube-api-access-97289\") pod \"be652fcd-c5e8-4086-a10a-f80ded1187f4\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " Oct 30 06:47:32.905576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905480 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-data\") pod \"567e2929-da18-45e4-a1e4-4e86f1ad1855\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " Oct 30 06:47:32.905576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905508 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-workload-socket\") pod \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " Oct 30 06:47:32.905576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905534 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-kube-api-access-cvkgm\" (UniqueName: \"kubernetes.io/projected/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-kube-api-access-cvkgm\") pod \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " Oct 30 06:47:32.905576 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905574 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-credential-socket\") pod \"be652fcd-c5e8-4086-a10a-f80ded1187f4\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905603 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-envoy\") pod \"567e2929-da18-45e4-a1e4-4e86f1ad1855\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905633 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be652fcd-c5e8-4086-a10a-f80ded1187f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be652fcd-c5e8-4086-a10a-f80ded1187f4-istiod-ca-cert\") pod \"be652fcd-c5e8-4086-a10a-f80ded1187f4\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905678 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-workload-certs\") pod \"be652fcd-c5e8-4086-a10a-f80ded1187f4\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905702 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-token\" (UniqueName: \"kubernetes.io/projected/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-token\") pod \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905729 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-workload-socket\") pod \"567e2929-da18-45e4-a1e4-4e86f1ad1855\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905761 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-workload-socket\") pod \"be652fcd-c5e8-4086-a10a-f80ded1187f4\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905810 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/567e2929-da18-45e4-a1e4-4e86f1ad1855-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567e2929-da18-45e4-a1e4-4e86f1ad1855-istiod-ca-cert\") pod \"567e2929-da18-45e4-a1e4-4e86f1ad1855\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905846 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-data\") pod \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905877 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-podinfo\") pod \"567e2929-da18-45e4-a1e4-4e86f1ad1855\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905933 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-token\" (UniqueName: \"kubernetes.io/projected/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-token\") pod \"be652fcd-c5e8-4086-a10a-f80ded1187f4\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905935 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-workload-socket" pod "c3934fa5-c92d-4efd-b675-e9764ab0fcfa" (UID: "c3934fa5-c92d-4efd-b675-e9764ab0fcfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905960 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-workload-certs\") pod \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.905993 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istiod-ca-cert\") pod \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.906021 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-envoy\") pod \"be652fcd-c5e8-4086-a10a-f80ded1187f4\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.906082 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-podinfo\") pod \"be652fcd-c5e8-4086-a10a-f80ded1187f4\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.906119 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-credential-socket\") pod \"567e2929-da18-45e4-a1e4-4e86f1ad1855\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.906145 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-workload-certs\") pod \"567e2929-da18-45e4-a1e4-4e86f1ad1855\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.906176 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-token\" (UniqueName: \"kubernetes.io/projected/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-token\") pod \"567e2929-da18-45e4-a1e4-4e86f1ad1855\" (UID: \"567e2929-da18-45e4-a1e4-4e86f1ad1855\") " Oct 30 06:47:32.906592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.906205 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-credential-socket\") pod \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\" (UID: \"c3934fa5-c92d-4efd-b675-e9764ab0fcfa\") " Oct 30 06:47:32.907686 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.906624 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-data\") pod \"be652fcd-c5e8-4086-a10a-f80ded1187f4\" (UID: \"be652fcd-c5e8-4086-a10a-f80ded1187f4\") " Oct 30 06:47:32.908262 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.908240 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:32.908436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.908420 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-data" pod "be652fcd-c5e8-4086-a10a-f80ded1187f4" (UID: "be652fcd-c5e8-4086-a10a-f80ded1187f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.910903 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.910856 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-envoy" pod "c3934fa5-c92d-4efd-b675-e9764ab0fcfa" (UID: "c3934fa5-c92d-4efd-b675-e9764ab0fcfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.911057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.911018 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567e2929-da18-45e4-a1e4-4e86f1ad1855-kube-api-access-n79gb" pod "567e2929-da18-45e4-a1e4-4e86f1ad1855" (UID: "567e2929-da18-45e4-a1e4-4e86f1ad1855"). InnerVolumeSpecName "kube-api-access-n79gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:32.911368 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.911343 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be652fcd-c5e8-4086-a10a-f80ded1187f4-istiod-ca-cert" pod "be652fcd-c5e8-4086-a10a-f80ded1187f4" (UID: "be652fcd-c5e8-4086-a10a-f80ded1187f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:32.913712 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.911628 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-data" pod "567e2929-da18-45e4-a1e4-4e86f1ad1855" (UID: "567e2929-da18-45e4-a1e4-4e86f1ad1855"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.914271 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.914036 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-workload-certs" pod "be652fcd-c5e8-4086-a10a-f80ded1187f4" (UID: "be652fcd-c5e8-4086-a10a-f80ded1187f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.914475 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.914452 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be652fcd-c5e8-4086-a10a-f80ded1187f4-kube-api-access-97289" pod "be652fcd-c5e8-4086-a10a-f80ded1187f4" (UID: "be652fcd-c5e8-4086-a10a-f80ded1187f4"). InnerVolumeSpecName "kube-api-access-97289". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:32.914595 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.914488 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-credential-socket" pod "be652fcd-c5e8-4086-a10a-f80ded1187f4" (UID: "be652fcd-c5e8-4086-a10a-f80ded1187f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.914732 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.914704 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-workload-socket" pod "be652fcd-c5e8-4086-a10a-f80ded1187f4" (UID: "be652fcd-c5e8-4086-a10a-f80ded1187f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.914859 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.914843 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-workload-socket" pod "567e2929-da18-45e4-a1e4-4e86f1ad1855" (UID: "567e2929-da18-45e4-a1e4-4e86f1ad1855"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.914926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.914892 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/567e2929-da18-45e4-a1e4-4e86f1ad1855-istiod-ca-cert" pod "567e2929-da18-45e4-a1e4-4e86f1ad1855" (UID: "567e2929-da18-45e4-a1e4-4e86f1ad1855"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:32.915117 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.915099 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-data" pod "c3934fa5-c92d-4efd-b675-e9764ab0fcfa" (UID: "c3934fa5-c92d-4efd-b675-e9764ab0fcfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.915473 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.915451 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istiod-ca-cert" pod "c3934fa5-c92d-4efd-b675-e9764ab0fcfa" (UID: "c3934fa5-c92d-4efd-b675-e9764ab0fcfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:32.915624 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.915598 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-token" pod "be652fcd-c5e8-4086-a10a-f80ded1187f4" (UID: "be652fcd-c5e8-4086-a10a-f80ded1187f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:32.915900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.915871 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-credential-socket" pod "567e2929-da18-45e4-a1e4-4e86f1ad1855" (UID: "567e2929-da18-45e4-a1e4-4e86f1ad1855"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.915987 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.915913 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-token" pod "c3934fa5-c92d-4efd-b675-e9764ab0fcfa" (UID: "c3934fa5-c92d-4efd-b675-e9764ab0fcfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:32.916137 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.916108 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-workload-certs" pod "c3934fa5-c92d-4efd-b675-e9764ab0fcfa" (UID: "c3934fa5-c92d-4efd-b675-e9764ab0fcfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.916296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.916264 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-workload-certs" pod "567e2929-da18-45e4-a1e4-4e86f1ad1855" (UID: "567e2929-da18-45e4-a1e4-4e86f1ad1855"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.917345 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.917319 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-credential-socket" pod "c3934fa5-c92d-4efd-b675-e9764ab0fcfa" (UID: "c3934fa5-c92d-4efd-b675-e9764ab0fcfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.918664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.918172 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-envoy" pod "567e2929-da18-45e4-a1e4-4e86f1ad1855" (UID: "567e2929-da18-45e4-a1e4-4e86f1ad1855"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.919775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.919755 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-envoy" pod "be652fcd-c5e8-4086-a10a-f80ded1187f4" (UID: "be652fcd-c5e8-4086-a10a-f80ded1187f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:32.919876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.919845 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-podinfo" pod "be652fcd-c5e8-4086-a10a-f80ded1187f4" (UID: "be652fcd-c5e8-4086-a10a-f80ded1187f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:32.920673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.920643 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-token" pod "567e2929-da18-45e4-a1e4-4e86f1ad1855" (UID: "567e2929-da18-45e4-a1e4-4e86f1ad1855"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:32.923605 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.923560 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-podinfo" pod "c3934fa5-c92d-4efd-b675-e9764ab0fcfa" (UID: "c3934fa5-c92d-4efd-b675-e9764ab0fcfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:32.923605 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.923581 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-kube-api-access-cvkgm" pod "c3934fa5-c92d-4efd-b675-e9764ab0fcfa" (UID: "c3934fa5-c92d-4efd-b675-e9764ab0fcfa"). InnerVolumeSpecName "kube-api-access-cvkgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:32.926439 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.925475 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-podinfo" pod "567e2929-da18-45e4-a1e4-4e86f1ad1855" (UID: "567e2929-da18-45e4-a1e4-4e86f1ad1855"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:32.927761 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:32.927741 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:47:33.008941 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.008854 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-kube-api-access-d7dj7\" (UniqueName: \"kubernetes.io/projected/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-kube-api-access-d7dj7\") pod \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " Oct 30 06:47:33.008941 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.008920 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-token\" (UniqueName: \"kubernetes.io/projected/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-token\") pod \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " Oct 30 06:47:33.009177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.009078 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-credential-socket\") pod \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " Oct 30 06:47:33.009177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.009112 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istiod-ca-cert\") pod \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " Oct 30 06:47:33.009347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.009247 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-data\") pod \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " Oct 30 06:47:33.009347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.009308 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-podinfo\") pod \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " Oct 30 06:47:33.009477 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.009348 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-workload-certs\") pod \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " Oct 30 06:47:33.009477 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.009400 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-envoy\") pod \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " Oct 30 06:47:33.009477 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.009434 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-workload-socket\") pod \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\" (UID: \"0bc0b89a-0fb5-4632-b6e2-d8586cd93435\") " Oct 30 06:47:33.009820 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.009480 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-credential-socket" pod "0bc0b89a-0fb5-4632-b6e2-d8586cd93435" (UID: "0bc0b89a-0fb5-4632-b6e2-d8586cd93435"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010105 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010135 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010151 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010165 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010178 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010191 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010239 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istiod-ca-cert" pod "0bc0b89a-0fb5-4632-b6e2-d8586cd93435" (UID: "0bc0b89a-0fb5-4632-b6e2-d8586cd93435"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010252 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n79gb\" (UniqueName: \"kubernetes.io/projected/567e2929-da18-45e4-a1e4-4e86f1ad1855-kube-api-access-n79gb\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010289 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010306 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010321 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97289\" (UniqueName: \"kubernetes.io/projected/be652fcd-c5e8-4086-a10a-f80ded1187f4-kube-api-access-97289\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010338 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010351 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010365 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvkgm\" (UniqueName: \"kubernetes.io/projected/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-kube-api-access-cvkgm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010377 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010390 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010500 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be652fcd-c5e8-4086-a10a-f80ded1187f4-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010513 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010525 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010538 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/567e2929-da18-45e4-a1e4-4e86f1ad1855-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010552 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010564 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/567e2929-da18-45e4-a1e4-4e86f1ad1855-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010577 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010589 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/567e2929-da18-45e4-a1e4-4e86f1ad1855-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010601 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010615 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010628 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3934fa5-c92d-4efd-b675-e9764ab0fcfa-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010641 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be652fcd-c5e8-4086-a10a-f80ded1187f4-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010470 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-workload-socket" pod "0bc0b89a-0fb5-4632-b6e2-d8586cd93435" (UID: "0bc0b89a-0fb5-4632-b6e2-d8586cd93435"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.010882 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.010844 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-data" pod "0bc0b89a-0fb5-4632-b6e2-d8586cd93435" (UID: "0bc0b89a-0fb5-4632-b6e2-d8586cd93435"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.012951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.012465 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-workload-certs" pod "0bc0b89a-0fb5-4632-b6e2-d8586cd93435" (UID: "0bc0b89a-0fb5-4632-b6e2-d8586cd93435"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.012951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.012770 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-podinfo" pod "0bc0b89a-0fb5-4632-b6e2-d8586cd93435" (UID: "0bc0b89a-0fb5-4632-b6e2-d8586cd93435"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:33.012951 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.012943 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-envoy" pod "0bc0b89a-0fb5-4632-b6e2-d8586cd93435" (UID: "0bc0b89a-0fb5-4632-b6e2-d8586cd93435"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:33.013760 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.013734 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-kube-api-access-d7dj7" pod "0bc0b89a-0fb5-4632-b6e2-d8586cd93435" (UID: "0bc0b89a-0fb5-4632-b6e2-d8586cd93435"). InnerVolumeSpecName "kube-api-access-d7dj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:33.014915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.014889 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-token" pod "0bc0b89a-0fb5-4632-b6e2-d8586cd93435" (UID: "0bc0b89a-0fb5-4632-b6e2-d8586cd93435"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:33.111896 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.111871 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.111896 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.111896 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.111896 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.111904 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.112112 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.111911 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.112112 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.111920 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.112112 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.111929 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.112112 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.111938 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7dj7\" (UniqueName: \"kubernetes.io/projected/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-kube-api-access-d7dj7\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.112112 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.111946 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bc0b89a-0fb5-4632-b6e2-d8586cd93435-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:33.149578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.147753 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:33.433037 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.433006 6999 generic.go:356] "Generic (PLEG): container finished" podID="be652fcd-c5e8-4086-a10a-f80ded1187f4" containerID="50e45b5fcb9fcdb880384d14609ab3560e3f623a88f16aeb7f7dcb47b6deb1a1" exitCode=0 Oct 30 06:47:33.433698 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.433070 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" Oct 30 06:47:33.433698 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.433062 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" event={"ID":"be652fcd-c5e8-4086-a10a-f80ded1187f4","Type":"ContainerDied","Data":"50e45b5fcb9fcdb880384d14609ab3560e3f623a88f16aeb7f7dcb47b6deb1a1"} Oct 30 06:47:33.433698 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.433268 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr" event={"ID":"be652fcd-c5e8-4086-a10a-f80ded1187f4","Type":"ContainerDied","Data":"5e9b8aaabb208db547efad0c85d6c66bdaed0e85604d1bd5774dba133104151b"} Oct 30 06:47:33.433698 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.433334 6999 scope.go:122] "RemoveContainer" containerID="50e45b5fcb9fcdb880384d14609ab3560e3f623a88f16aeb7f7dcb47b6deb1a1" Oct 30 06:47:33.437979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.437957 6999 generic.go:356] "Generic (PLEG): container finished" podID="0bc0b89a-0fb5-4632-b6e2-d8586cd93435" containerID="c520d227fef6f43bb0c39ff7df673d827acfab5fe3ad7bb225fac714cf2812cf" exitCode=0 Oct 30 06:47:33.438126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.438113 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" Oct 30 06:47:33.438205 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.438140 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" event={"ID":"0bc0b89a-0fb5-4632-b6e2-d8586cd93435","Type":"ContainerDied","Data":"c520d227fef6f43bb0c39ff7df673d827acfab5fe3ad7bb225fac714cf2812cf"} Oct 30 06:47:33.438676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.438653 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj" event={"ID":"0bc0b89a-0fb5-4632-b6e2-d8586cd93435","Type":"ContainerDied","Data":"60db96c3b755fdfdfaf457e26bdb5badf3770d13b1df45ff8d44ead6d802e94f"} Oct 30 06:47:33.440536 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.440517 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" Oct 30 06:47:33.440536 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.440527 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" event={"ID":"c3934fa5-c92d-4efd-b675-e9764ab0fcfa","Type":"ContainerDied","Data":"9d8f492ab96026ef5f521527935f78c2771cb36afc05906fdbbe496443d5641f"} Oct 30 06:47:33.443260 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.443200 6999 generic.go:356] "Generic (PLEG): container finished" podID="567e2929-da18-45e4-a1e4-4e86f1ad1855" containerID="8d8b4768453bc5d1391d988e6b836c8fa81e9d69885ed9f432da054567e0fa66" exitCode=0 Oct 30 06:47:33.443415 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.443400 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" Oct 30 06:47:33.443991 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.443705 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" event={"ID":"567e2929-da18-45e4-a1e4-4e86f1ad1855","Type":"ContainerDied","Data":"8d8b4768453bc5d1391d988e6b836c8fa81e9d69885ed9f432da054567e0fa66"} Oct 30 06:47:33.443991 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.443754 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4" event={"ID":"567e2929-da18-45e4-a1e4-4e86f1ad1855","Type":"ContainerDied","Data":"89e6b330929926adcffae8621324671ba8d4e11803aa0f7778de187382c1c19f"} Oct 30 06:47:33.447349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.447042 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" event={"ID":"c3e004f9-f184-4b09-835a-b314650513d4","Type":"ContainerDied","Data":"7de2e0ea2ca9e89f25f3e11739a8d0b4acaf9e6bb6c8bf881d44118121c60379"} Oct 30 06:47:33.447349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.447153 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" Oct 30 06:47:33.458238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.458122 6999 scope.go:122] "RemoveContainer" containerID="50e45b5fcb9fcdb880384d14609ab3560e3f623a88f16aeb7f7dcb47b6deb1a1" Oct 30 06:47:33.458521 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:47:33.458487 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50e45b5fcb9fcdb880384d14609ab3560e3f623a88f16aeb7f7dcb47b6deb1a1\": container with ID starting with 50e45b5fcb9fcdb880384d14609ab3560e3f623a88f16aeb7f7dcb47b6deb1a1 not found: ID does not exist" containerID="50e45b5fcb9fcdb880384d14609ab3560e3f623a88f16aeb7f7dcb47b6deb1a1" Oct 30 06:47:33.458619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.458526 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50e45b5fcb9fcdb880384d14609ab3560e3f623a88f16aeb7f7dcb47b6deb1a1"} err="failed to get container status \"50e45b5fcb9fcdb880384d14609ab3560e3f623a88f16aeb7f7dcb47b6deb1a1\": rpc error: code = NotFound desc = could not find container \"50e45b5fcb9fcdb880384d14609ab3560e3f623a88f16aeb7f7dcb47b6deb1a1\": container with ID starting with 50e45b5fcb9fcdb880384d14609ab3560e3f623a88f16aeb7f7dcb47b6deb1a1 not found: ID does not exist" Oct 30 06:47:33.458619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.458547 6999 scope.go:122] "RemoveContainer" containerID="c520d227fef6f43bb0c39ff7df673d827acfab5fe3ad7bb225fac714cf2812cf" Oct 30 06:47:33.470547 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.470521 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:47:33.470699 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.470575 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r" podUID="c3e004f9-f184-4b09-835a-b314650513d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:47:33.479289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.479272 6999 scope.go:122] "RemoveContainer" containerID="c520d227fef6f43bb0c39ff7df673d827acfab5fe3ad7bb225fac714cf2812cf" Oct 30 06:47:33.479591 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:47:33.479572 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c520d227fef6f43bb0c39ff7df673d827acfab5fe3ad7bb225fac714cf2812cf\": container with ID starting with c520d227fef6f43bb0c39ff7df673d827acfab5fe3ad7bb225fac714cf2812cf not found: ID does not exist" containerID="c520d227fef6f43bb0c39ff7df673d827acfab5fe3ad7bb225fac714cf2812cf" Oct 30 06:47:33.479685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.479602 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c520d227fef6f43bb0c39ff7df673d827acfab5fe3ad7bb225fac714cf2812cf"} err="failed to get container status \"c520d227fef6f43bb0c39ff7df673d827acfab5fe3ad7bb225fac714cf2812cf\": rpc error: code = NotFound desc = could not find container \"c520d227fef6f43bb0c39ff7df673d827acfab5fe3ad7bb225fac714cf2812cf\": container with ID starting with c520d227fef6f43bb0c39ff7df673d827acfab5fe3ad7bb225fac714cf2812cf not found: ID does not exist" Oct 30 06:47:33.479685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.479624 6999 scope.go:122] "RemoveContainer" containerID="714ae399d71b09d70ec1fcf5f637b14425595ff8f0621c56baed7f9e7ba64d20" Oct 30 06:47:33.500312 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.500292 6999 scope.go:122] "RemoveContainer" containerID="8d8b4768453bc5d1391d988e6b836c8fa81e9d69885ed9f432da054567e0fa66" Oct 30 06:47:33.513322 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.512378 6999 scope.go:122] "RemoveContainer" containerID="8d8b4768453bc5d1391d988e6b836c8fa81e9d69885ed9f432da054567e0fa66" Oct 30 06:47:33.513322 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:47:33.513129 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d8b4768453bc5d1391d988e6b836c8fa81e9d69885ed9f432da054567e0fa66\": container with ID starting with 8d8b4768453bc5d1391d988e6b836c8fa81e9d69885ed9f432da054567e0fa66 not found: ID does not exist" containerID="8d8b4768453bc5d1391d988e6b836c8fa81e9d69885ed9f432da054567e0fa66" Oct 30 06:47:33.513322 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.513168 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8b4768453bc5d1391d988e6b836c8fa81e9d69885ed9f432da054567e0fa66"} err="failed to get container status \"8d8b4768453bc5d1391d988e6b836c8fa81e9d69885ed9f432da054567e0fa66\": rpc error: code = NotFound desc = could not find container \"8d8b4768453bc5d1391d988e6b836c8fa81e9d69885ed9f432da054567e0fa66\": container with ID starting with 8d8b4768453bc5d1391d988e6b836c8fa81e9d69885ed9f432da054567e0fa66 not found: ID does not exist" Oct 30 06:47:33.513322 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.513192 6999 scope.go:122] "RemoveContainer" containerID="dcd27fccf95c79d87b5b36d4046ae1f19e8293551800b155b0c77fc718e6cc5f" Oct 30 06:47:33.541356 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.538265 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr"] Oct 30 06:47:33.552265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.552245 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-sb7zr"] Oct 30 06:47:33.591945 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.591923 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4"] Oct 30 06:47:33.595448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.595428 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-tb7w4"] Oct 30 06:47:33.602756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.602732 6999 patch_prober.go:29] interesting pod/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:47:33.602864 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.602786 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9" podUID="c3934fa5-c92d-4efd-b675-e9764ab0fcfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:47:33.611732 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.611712 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="567e2929-da18-45e4-a1e4-4e86f1ad1855" path="/var/lib/kubelet/pods/567e2929-da18-45e4-a1e4-4e86f1ad1855/volumes" Oct 30 06:47:33.612249 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.612235 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be652fcd-c5e8-4086-a10a-f80ded1187f4" path="/var/lib/kubelet/pods/be652fcd-c5e8-4086-a10a-f80ded1187f4/volumes" Oct 30 06:47:33.630148 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.630127 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r"] Oct 30 06:47:33.634973 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.634951 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-6gz6r"] Oct 30 06:47:33.670022 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.669993 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj"] Oct 30 06:47:33.673104 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.673087 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a0083e-openshift-default-5bb4897bcc-g4mrj"] Oct 30 06:47:33.710208 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.710131 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9"] Oct 30 06:47:33.712988 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:33.712967 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-69f3fa-openshift-default-6cb89794d5-ddhg9"] Oct 30 06:47:34.171262 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:47:34.167939 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:47:34.183113 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:34.183081 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:34.322960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:34.322933 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:35.001115 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:35.001076 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:35.149778 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:35.149742 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:35.294236 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:35.294133 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:35.610146 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:35.610116 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bc0b89a-0fb5-4632-b6e2-d8586cd93435" path="/var/lib/kubelet/pods/0bc0b89a-0fb5-4632-b6e2-d8586cd93435/volumes" Oct 30 06:47:35.610667 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:35.610653 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3934fa5-c92d-4efd-b675-e9764ab0fcfa" path="/var/lib/kubelet/pods/c3934fa5-c92d-4efd-b675-e9764ab0fcfa/volumes" Oct 30 06:47:35.611133 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:35.611119 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e004f9-f184-4b09-835a-b314650513d4" path="/var/lib/kubelet/pods/c3e004f9-f184-4b09-835a-b314650513d4/volumes" Oct 30 06:47:36.773345 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:36.773314 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:38.641117 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:38.641077 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:40.602363 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:40.602328 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t"] Oct 30 06:47:40.602872 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:40.602517 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" podUID="0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" containerName="istio-proxy" containerID="cri-o://d7ee6de12adec7c4f813f2a858fa0e72c7c825560be9be442ca77f40223e0294" gracePeriod=30 Oct 30 06:47:41.785993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:41.785953 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:42.005735 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:42.005698 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:42.148853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:42.148816 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:42.310345 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:42.310315 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:42.500294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:42.499351 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:42.998355 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:42.997722 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:43.675239 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:43.671597 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:44.492776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.492737 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw"] Oct 30 06:47:44.493751 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493729 6999 cpu_manager.go:414] "containerMap: removing container" podUID="c3934fa5-c92d-4efd-b675-e9764ab0fcfa" containerName="istio-proxy" Oct 30 06:47:44.493751 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493750 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3934fa5-c92d-4efd-b675-e9764ab0fcfa" containerName="istio-proxy" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493766 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b253e877-6873-4e91-a50b-30fef43bcd3e" containerName="istio-proxy" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493771 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b253e877-6873-4e91-a50b-30fef43bcd3e" containerName="istio-proxy" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493785 6999 cpu_manager.go:414] "containerMap: removing container" podUID="0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2" containerName="discovery" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493791 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2" containerName="discovery" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493807 6999 cpu_manager.go:414] "containerMap: removing container" podUID="0bc0b89a-0fb5-4632-b6e2-d8586cd93435" containerName="istio-proxy" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493812 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bc0b89a-0fb5-4632-b6e2-d8586cd93435" containerName="istio-proxy" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493818 6999 cpu_manager.go:414] "containerMap: removing container" podUID="567e2929-da18-45e4-a1e4-4e86f1ad1855" containerName="istio-proxy" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493822 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="567e2929-da18-45e4-a1e4-4e86f1ad1855" containerName="istio-proxy" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493851 6999 cpu_manager.go:414] "containerMap: removing container" podUID="86cc6726-5087-432a-8487-5dd10dfb4c51" containerName="istio-proxy" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493856 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86cc6726-5087-432a-8487-5dd10dfb4c51" containerName="istio-proxy" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493862 6999 cpu_manager.go:414] "containerMap: removing container" podUID="62421fb5-a574-4d99-a8e8-747f202e7e24" containerName="istio-proxy" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493868 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62421fb5-a574-4d99-a8e8-747f202e7e24" containerName="istio-proxy" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493879 6999 cpu_manager.go:414] "containerMap: removing container" podUID="be652fcd-c5e8-4086-a10a-f80ded1187f4" containerName="istio-proxy" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493883 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be652fcd-c5e8-4086-a10a-f80ded1187f4" containerName="istio-proxy" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493891 6999 cpu_manager.go:414] "containerMap: removing container" podUID="c3e004f9-f184-4b09-835a-b314650513d4" containerName="istio-proxy" Oct 30 06:47:44.493961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493895 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e004f9-f184-4b09-835a-b314650513d4" containerName="istio-proxy" Oct 30 06:47:44.494853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.493991 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="86cc6726-5087-432a-8487-5dd10dfb4c51" containerName="istio-proxy" Oct 30 06:47:44.494853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.494001 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="62421fb5-a574-4d99-a8e8-747f202e7e24" containerName="istio-proxy" Oct 30 06:47:44.494853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.494008 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="be652fcd-c5e8-4086-a10a-f80ded1187f4" containerName="istio-proxy" Oct 30 06:47:44.494853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.494014 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e004f9-f184-4b09-835a-b314650513d4" containerName="istio-proxy" Oct 30 06:47:44.494853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.494022 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3934fa5-c92d-4efd-b675-e9764ab0fcfa" containerName="istio-proxy" Oct 30 06:47:44.494853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.494029 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b253e877-6873-4e91-a50b-30fef43bcd3e" containerName="istio-proxy" Oct 30 06:47:44.494853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.494034 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8eee62-2ff4-4caa-a795-79ae5e1ffcd2" containerName="discovery" Oct 30 06:47:44.494853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.494040 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bc0b89a-0fb5-4632-b6e2-d8586cd93435" containerName="istio-proxy" Oct 30 06:47:44.494853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.494047 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="567e2929-da18-45e4-a1e4-4e86f1ad1855" containerName="istio-proxy" Oct 30 06:47:44.498820 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.498789 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:44.518811 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.518781 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw"] Oct 30 06:47:44.575699 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.575670 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-b66aad-serving-cert\"" Oct 30 06:47:44.599919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.599888 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cf76c9b-fa0c-4575-aeae-c14e7487809b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:44.600119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.600017 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-metrics-certs\") pod \"maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:44.600119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.600075 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-maas-api-tls\") pod \"maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:44.600244 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.600164 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzj68\" (UniqueName: \"kubernetes.io/projected/9cf76c9b-fa0c-4575-aeae-c14e7487809b-kube-api-access-xzj68\") pod \"maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:44.620224 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.620186 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:44.703004 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.702966 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-maas-api-tls\") pod \"maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:44.703311 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.703285 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzj68\" (UniqueName: \"kubernetes.io/projected/9cf76c9b-fa0c-4575-aeae-c14e7487809b-kube-api-access-xzj68\") pod \"maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:44.703804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.703784 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cf76c9b-fa0c-4575-aeae-c14e7487809b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:44.705739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.704065 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-metrics-certs\") pod \"maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:44.705739 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:47:44.704083 6999 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-serving-cert: secret "maas-api-e2e-inf-b-b66aad-serving-cert" not found Oct 30 06:47:44.705739 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:47:44.704175 6999 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-maas-api-tls podName:9cf76c9b-fa0c-4575-aeae-c14e7487809b nodeName:}" failed. No retries permitted until 2026-10-30 06:47:45.204145149 +0000 UTC m=+3231.713165028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-maas-api-tls") pod "maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" (UID: "9cf76c9b-fa0c-4575-aeae-c14e7487809b") : secret "maas-api-e2e-inf-b-b66aad-serving-cert" not found Oct 30 06:47:44.705739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.705677 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cf76c9b-fa0c-4575-aeae-c14e7487809b-openshift-service-ca\") pod \"maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:44.726403 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.726372 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-metrics-certs\") pod \"maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:44.729294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.729255 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzj68\" (UniqueName: \"kubernetes.io/projected/9cf76c9b-fa0c-4575-aeae-c14e7487809b-kube-api-access-xzj68\") pod \"maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:44.772396 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:44.772305 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:45.139122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:45.139087 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:45.209833 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:45.209795 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-maas-api-tls\") pod \"maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:45.212690 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:45.212663 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-maas-api-tls\") pod \"maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:45.416249 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:45.416143 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:45.732075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:45.731924 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw"] Oct 30 06:47:45.732473 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:45.732186 6999 patch_prober.go:29] interesting pod/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 06:47:45.732473 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:45.732247 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" podUID="0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 06:47:45.733125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:45.733106 6999 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:47:45.936735 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:45.936711 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:47:46.026002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.025914 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-workload-socket\") pod \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " Oct 30 06:47:46.026189 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.026063 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-podinfo\") pod \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " Oct 30 06:47:46.026189 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.026114 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-kube-api-access-pqbsq\" (UniqueName: \"kubernetes.io/projected/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-kube-api-access-pqbsq\") pod \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " Oct 30 06:47:46.026377 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.026185 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-data\") pod \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " Oct 30 06:47:46.026377 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.026273 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-workload-socket" pod "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" (UID: "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:46.026377 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.026316 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-workload-certs\") pod \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " Oct 30 06:47:46.026377 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.026370 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-token\" (UniqueName: \"kubernetes.io/projected/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-token\") pod \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " Oct 30 06:47:46.026654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.026418 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istiod-ca-cert\") pod \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " Oct 30 06:47:46.026654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.026463 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-credential-socket\") pod \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " Oct 30 06:47:46.026654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.026555 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-envoy\") pod \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\" (UID: \"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d\") " Oct 30 06:47:46.026654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.026602 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-data" pod "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" (UID: "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:46.026921 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.026780 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-credential-socket" pod "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" (UID: "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:46.026921 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.026830 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-workload-certs" pod "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" (UID: "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:46.027248 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.027208 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:46.027557 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.027254 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:46.027557 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.027269 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:46.027557 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.027282 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:46.027557 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.027403 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istiod-ca-cert" pod "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" (UID: "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:47:46.028985 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.028961 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-podinfo" pod "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" (UID: "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:47:46.029521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.029492 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-envoy" pod "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" (UID: "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:47:46.029638 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.029547 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-token" pod "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" (UID: "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:46.029692 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.029652 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-kube-api-access-pqbsq" pod "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" (UID: "0f3fb8b1-0bba-4b12-bc51-d9f137132e4d"). InnerVolumeSpecName "kube-api-access-pqbsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:47:46.128162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.128136 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqbsq\" (UniqueName: \"kubernetes.io/projected/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-kube-api-access-pqbsq\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:46.128305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.128166 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:46.128305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.128180 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:46.128305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.128191 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:46.128305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.128202 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:47:46.594303 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.594259 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" event={"ID":"9cf76c9b-fa0c-4575-aeae-c14e7487809b","Type":"ContainerStarted","Data":"981c4edb59f0defddc66ffd1764ce076a41a7d893f50a7123af0846f8fa28aaa"} Oct 30 06:47:46.594510 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.594343 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" event={"ID":"9cf76c9b-fa0c-4575-aeae-c14e7487809b","Type":"ContainerStarted","Data":"ecec1ec0befe9da6ab142a11576dfa1c24b897da4a295b829a24c0af0fc074d6"} Oct 30 06:47:46.594510 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.594424 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:46.595792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.595765 6999 generic.go:356] "Generic (PLEG): container finished" podID="0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" containerID="d7ee6de12adec7c4f813f2a858fa0e72c7c825560be9be442ca77f40223e0294" exitCode=0 Oct 30 06:47:46.595947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.595808 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" Oct 30 06:47:46.595947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.595856 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" event={"ID":"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d","Type":"ContainerDied","Data":"d7ee6de12adec7c4f813f2a858fa0e72c7c825560be9be442ca77f40223e0294"} Oct 30 06:47:46.595947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.595915 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t" event={"ID":"0f3fb8b1-0bba-4b12-bc51-d9f137132e4d","Type":"ContainerDied","Data":"af07b15bcd59ee816429f1612931810f67a9e2dd4390b6abbc58a5ee24d84abc"} Oct 30 06:47:46.595947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.595944 6999 scope.go:122] "RemoveContainer" containerID="d7ee6de12adec7c4f813f2a858fa0e72c7c825560be9be442ca77f40223e0294" Oct 30 06:47:46.610109 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.610092 6999 scope.go:122] "RemoveContainer" containerID="d7ee6de12adec7c4f813f2a858fa0e72c7c825560be9be442ca77f40223e0294" Oct 30 06:47:46.610372 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:47:46.610353 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ee6de12adec7c4f813f2a858fa0e72c7c825560be9be442ca77f40223e0294\": container with ID starting with d7ee6de12adec7c4f813f2a858fa0e72c7c825560be9be442ca77f40223e0294 not found: ID does not exist" containerID="d7ee6de12adec7c4f813f2a858fa0e72c7c825560be9be442ca77f40223e0294" Oct 30 06:47:46.610498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.610381 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ee6de12adec7c4f813f2a858fa0e72c7c825560be9be442ca77f40223e0294"} err="failed to get container status \"d7ee6de12adec7c4f813f2a858fa0e72c7c825560be9be442ca77f40223e0294\": rpc error: code = NotFound desc = could not find container \"d7ee6de12adec7c4f813f2a858fa0e72c7c825560be9be442ca77f40223e0294\": container with ID starting with d7ee6de12adec7c4f813f2a858fa0e72c7c825560be9be442ca77f40223e0294 not found: ID does not exist" Oct 30 06:47:46.624320 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.624288 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:46.639013 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.638962 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" podStartSLOduration=2.268043724 podStartE2EDuration="2.638944014s" totalImagesPullingTime="370.90029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:47:46.638029685 +0000 UTC m=+3233.147049573" watchObservedRunningTime="2026-10-30 06:47:46.638944014 +0000 UTC m=+3233.147963903" Oct 30 06:47:46.678182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.678139 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t"] Oct 30 06:47:46.688135 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:46.688107 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8d6349-openshift-default-6b9c99ddb7-pmx7t"] Oct 30 06:47:47.611752 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:47.611718 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" path="/var/lib/kubelet/pods/0f3fb8b1-0bba-4b12-bc51-d9f137132e4d/volumes" Oct 30 06:47:47.963315 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:47.963093 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:48.113329 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:48.113298 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:48.759471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:48.759439 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:49.674607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:49.674566 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:49.824722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:49.824686 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:50.615845 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:50.615813 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:52.397494 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:52.397464 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:52.611942 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:52.611915 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:47:54.142162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:54.142131 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:55.971519 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:55.971436 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:57.716188 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:57.716136 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:47:59.437145 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:47:59.437079 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:01.222522 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:01.222485 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:02.937258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:02.937202 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:04.143300 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:48:04.143204 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:48:04.701548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:04.701522 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:05.021011 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:05.020917 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:06.639703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:06.639616 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:07.610834 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:07.610795 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:08.262473 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.262433 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5"] Oct 30 06:48:08.263138 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.263121 6999 cpu_manager.go:414] "containerMap: removing container" podUID="0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" containerName="istio-proxy" Oct 30 06:48:08.263196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.263140 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" containerName="istio-proxy" Oct 30 06:48:08.263299 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.263289 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3fb8b1-0bba-4b12-bc51-d9f137132e4d" containerName="istio-proxy" Oct 30 06:48:08.269140 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.269119 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:08.302268 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.296404 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-62efb8-serving-cert\"" Oct 30 06:48:08.302268 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.301359 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5"] Oct 30 06:48:08.393512 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.393469 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:08.433659 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.433614 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dce711f-1134-46f0-ace9-ff3080839ede-maas-api-tls\") pod \"maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:08.433844 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.433728 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dce711f-1134-46f0-ace9-ff3080839ede-openshift-service-ca\") pod \"maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:08.433921 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.433904 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dce711f-1134-46f0-ace9-ff3080839ede-metrics-certs\") pod \"maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:08.433967 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.433940 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb5hs\" (UniqueName: \"kubernetes.io/projected/7dce711f-1134-46f0-ace9-ff3080839ede-kube-api-access-pb5hs\") pod \"maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:08.534862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.534781 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dce711f-1134-46f0-ace9-ff3080839ede-maas-api-tls\") pod \"maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:08.534862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.534849 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dce711f-1134-46f0-ace9-ff3080839ede-openshift-service-ca\") pod \"maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:08.535100 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.535001 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dce711f-1134-46f0-ace9-ff3080839ede-metrics-certs\") pod \"maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:08.535100 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.535035 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb5hs\" (UniqueName: \"kubernetes.io/projected/7dce711f-1134-46f0-ace9-ff3080839ede-kube-api-access-pb5hs\") pod \"maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:08.535665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.535639 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dce711f-1134-46f0-ace9-ff3080839ede-openshift-service-ca\") pod \"maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:08.537732 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.537705 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dce711f-1134-46f0-ace9-ff3080839ede-maas-api-tls\") pod \"maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:08.538954 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.538924 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:08.540765 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.540743 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dce711f-1134-46f0-ace9-ff3080839ede-metrics-certs\") pod \"maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:08.567850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.567818 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb5hs\" (UniqueName: \"kubernetes.io/projected/7dce711f-1134-46f0-ace9-ff3080839ede-kube-api-access-pb5hs\") pod \"maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:08.584936 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.584909 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:08.896810 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:08.896784 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5"] Oct 30 06:48:08.896941 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:48:08.896914 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dce711f_1134_46f0_ace9_ff3080839ede.slice/crio-55e49249a37ffeb49df648217259cf88ca90867ff50b0ea25707482bc3a8b1de WatchSource:0}: Error finding container 55e49249a37ffeb49df648217259cf88ca90867ff50b0ea25707482bc3a8b1de: Status 404 returned error can't find the container with id 55e49249a37ffeb49df648217259cf88ca90867ff50b0ea25707482bc3a8b1de Oct 30 06:48:09.759389 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:09.759355 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" event={"ID":"7dce711f-1134-46f0-ace9-ff3080839ede","Type":"ContainerStarted","Data":"12aa9e97abe4abc42ce31da04f5564a4b221b62b2ac3dd352d92dfb07d2c6d7d"} Oct 30 06:48:09.759816 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:09.759409 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" event={"ID":"7dce711f-1134-46f0-ace9-ff3080839ede","Type":"ContainerStarted","Data":"55e49249a37ffeb49df648217259cf88ca90867ff50b0ea25707482bc3a8b1de"} Oct 30 06:48:09.759816 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:09.759481 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:09.804068 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:09.804006 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" podStartSLOduration=1.363103819 podStartE2EDuration="1.803988394s" totalImagesPullingTime="440.884575ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:48:09.800596701 +0000 UTC m=+3256.309616601" watchObservedRunningTime="2026-10-30 06:48:09.803988394 +0000 UTC m=+3256.313008282" Oct 30 06:48:10.321060 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:10.321029 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:12.117966 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:12.117924 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:13.883983 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:13.883951 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:15.762568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:15.762532 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:15.771952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:15.771903 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:48:17.400645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:17.400611 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:17.541335 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:17.541302 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:17.695720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:17.695438 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:17.837068 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:17.837039 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:19.231068 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:19.231037 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:19.407083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:19.407050 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:19.645420 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:19.645394 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:19.812173 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:19.812139 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:19.958527 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:19.958443 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:20.105911 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:20.105875 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:20.503970 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:20.503936 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:20.647709 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:20.647676 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:21.621111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:21.621068 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:21.872849 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:21.872760 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:22.015784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:22.015752 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:22.806522 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:22.806489 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:23.565748 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:23.565717 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:23.896819 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:23.896793 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:24.860769 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:24.860731 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:25.458625 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:25.458592 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:25.960830 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:25.960802 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:27.249696 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:27.249665 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:27.869583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:27.869544 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:28.013915 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:28.013884 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:28.318565 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:28.318477 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:28.510386 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:28.510350 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:29.052947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:29.052914 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:29.213871 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:29.213843 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:30.745449 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:30.745418 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:32.482782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:32.482750 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:34.143083 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:48:34.143043 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:48:34.329742 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:34.329709 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:35.371034 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:35.370997 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:35.522816 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:35.522787 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:35.901633 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:35.901601 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:36.117234 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:36.117186 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:36.259292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:36.259190 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:36.555932 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:36.555850 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:36.857142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:36.857065 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:38.283158 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:38.283118 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:40.102943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:40.102909 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:40.320537 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:40.320506 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:40.466159 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:40.466123 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:42.032273 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:42.032242 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:42.176484 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:42.176443 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:42.384626 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:42.384592 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:42.972693 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:42.972660 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:43.887383 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:43.887230 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw"] Oct 30 06:48:43.908150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:43.908117 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw"] Oct 30 06:48:43.908562 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:43.908544 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:43.936439 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:43.936419 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1c148a\"/\"default-dockercfg-f89v8\"" Oct 30 06:48:43.936571 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:43.936507 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1c148a\"/\"test-model-1c148a-kserve-self-signed-certs\"" Oct 30 06:48:43.936614 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:43.936565 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1c148a\"/\"openshift-service-ca.crt\"" Oct 30 06:48:43.936614 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:43.936582 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1c148a\"/\"kube-root-ca.crt\"" Oct 30 06:48:43.951820 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:43.951788 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:43.999310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:43.999277 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784f9ba2-3d6b-4cf4-8546-32c975d271ed-tls-certs\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:43.999462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:43.999318 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwhkf\" (UniqueName: \"kubernetes.io/projected/784f9ba2-3d6b-4cf4-8546-32c975d271ed-kube-api-access-dwhkf\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:43.999519 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:43.999466 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-dshm\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:43.999566 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:43.999527 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-home\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:43.999675 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:43.999658 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-model-cache\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:43.999714 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:43.999693 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-tmp-dir\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:44.100902 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:44.100860 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784f9ba2-3d6b-4cf4-8546-32c975d271ed-tls-certs\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:44.101850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:44.101820 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwhkf\" (UniqueName: \"kubernetes.io/projected/784f9ba2-3d6b-4cf4-8546-32c975d271ed-kube-api-access-dwhkf\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:44.102501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:44.102479 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-dshm\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:44.102722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:44.102680 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-home\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:44.102851 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:44.102766 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-model-cache\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:44.102851 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:44.102798 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-tmp-dir\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:44.103540 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:44.103519 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-tmp-dir\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:44.109260 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:44.104286 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-home\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:44.109736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:44.109709 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-model-cache\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:44.136989 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:44.136959 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784f9ba2-3d6b-4cf4-8546-32c975d271ed-tls-certs\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:44.151617 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:44.151541 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-dshm\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:44.156047 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:44.156018 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwhkf\" (UniqueName: \"kubernetes.io/projected/784f9ba2-3d6b-4cf4-8546-32c975d271ed-kube-api-access-dwhkf\") pod \"test-model-1c148a-kserve-5d777bdb99-w2cfw\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:44.236988 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:44.236955 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:44.241166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:44.241142 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:44.566621 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:48:44.566588 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod784f9ba2_3d6b_4cf4_8546_32c975d271ed.slice/crio-5a653716665a76dc318c001600bf0160db5f865ae8ba85dcc21b299ae0bc1f57 WatchSource:0}: Error finding container 5a653716665a76dc318c001600bf0160db5f865ae8ba85dcc21b299ae0bc1f57: Status 404 returned error can't find the container with id 5a653716665a76dc318c001600bf0160db5f865ae8ba85dcc21b299ae0bc1f57 Oct 30 06:48:44.566789 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:44.566765 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw"] Oct 30 06:48:45.009865 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:45.009826 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" event={"ID":"784f9ba2-3d6b-4cf4-8546-32c975d271ed","Type":"ContainerStarted","Data":"dccc539411176d80990278b1b7176980fa3bb5cb91f65aa2adf5d179f1c129ab"} Oct 30 06:48:45.010350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:45.009878 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" event={"ID":"784f9ba2-3d6b-4cf4-8546-32c975d271ed","Type":"ContainerStarted","Data":"5a653716665a76dc318c001600bf0160db5f865ae8ba85dcc21b299ae0bc1f57"} Oct 30 06:48:45.054324 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:45.054265 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" podStartSLOduration=2.054251217 podStartE2EDuration="2.054251217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:48:45.051716816 +0000 UTC m=+3291.560736705" watchObservedRunningTime="2026-10-30 06:48:45.054251217 +0000 UTC m=+3291.563271105" Oct 30 06:48:45.237078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:45.237049 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:46.140536 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:46.140507 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:48.166816 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:48.166779 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:50.032902 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:50.032867 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:51.811188 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:51.811145 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:53.711909 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:53.711874 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:54.238177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:54.238133 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:54.250967 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:54.250939 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:55.100397 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:55.100315 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:48:55.676333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:55.676290 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:57.410247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:57.410197 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:48:59.183763 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:48:59.183732 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:01.015460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:01.015430 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:02.908600 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:02.908550 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:03.190490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:03.190407 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:03.591549 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:03.591462 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:04.109591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:04.109539 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:04.145239 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:49:04.145188 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:49:04.643275 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:04.643188 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:04.882128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:04.882091 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:05.052204 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:05.052096 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:05.225606 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:05.225578 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:06.835204 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:06.835169 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:07.732260 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:07.732194 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:08.961592 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:08.961564 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:09.132252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:09.132182 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:09.283710 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:09.283637 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:10.791963 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:10.791931 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:10.942659 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:10.942620 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:12.666366 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:12.666325 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:13.468267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:13.468207 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:14.451409 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:14.451377 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:16.231073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:16.231040 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:18.017477 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:18.017442 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:19.780655 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:19.780621 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:21.636100 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:21.636072 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:23.324745 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:23.324709 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:25.128102 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:25.128017 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:26.890541 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:26.890511 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:28.685764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:28.685721 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:30.628953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:30.628916 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:32.471895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.471860 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp"] Oct 30 06:49:32.476561 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.476542 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:32.495452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.495429 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8baf7e-serving-cert\"" Oct 30 06:49:32.512309 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.512279 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp"] Oct 30 06:49:32.544020 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.543995 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-metrics-certs\") pod \"maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:32.544171 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.544028 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bg5w\" (UniqueName: \"kubernetes.io/projected/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-kube-api-access-8bg5w\") pod \"maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:32.544294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.544267 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-maas-api-tls\") pod \"maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:32.544430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.544410 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:32.605433 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.605401 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:32.645798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.645764 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-metrics-certs\") pod \"maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:32.645798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.645800 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bg5w\" (UniqueName: \"kubernetes.io/projected/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-kube-api-access-8bg5w\") pod \"maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:32.646064 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.646018 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-maas-api-tls\") pod \"maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:32.646403 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.646201 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:32.646939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.646919 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:32.648675 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.648652 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-maas-api-tls\") pod \"maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:32.648675 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.648664 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-metrics-certs\") pod \"maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:32.669866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.669842 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bg5w\" (UniqueName: \"kubernetes.io/projected/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-kube-api-access-8bg5w\") pod \"maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:32.753154 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.753050 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:32.791034 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.790999 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:32.901957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:32.898768 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:33.106340 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:49:33.106310 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b70aac_3036_44cb_ae26_d4e0bea0a45e.slice/crio-5219123d04b8dfc6bbbb86be0da8068e7ceb5ff97e669b6ffd3b1942dd36a253 WatchSource:0}: Error finding container 5219123d04b8dfc6bbbb86be0da8068e7ceb5ff97e669b6ffd3b1942dd36a253: Status 404 returned error can't find the container with id 5219123d04b8dfc6bbbb86be0da8068e7ceb5ff97e669b6ffd3b1942dd36a253 Oct 30 06:49:33.109784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:33.109759 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp"] Oct 30 06:49:33.366424 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:33.366386 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" event={"ID":"e7b70aac-3036-44cb-ae26-d4e0bea0a45e","Type":"ContainerStarted","Data":"5219123d04b8dfc6bbbb86be0da8068e7ceb5ff97e669b6ffd3b1942dd36a253"} Oct 30 06:49:34.143345 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:49:34.143304 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:49:34.375055 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:34.375019 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" event={"ID":"e7b70aac-3036-44cb-ae26-d4e0bea0a45e","Type":"ContainerStarted","Data":"59b374922932d63b6fc83bf852d5f391d5512039c32af75ac387cf330cf471e6"} Oct 30 06:49:34.375292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:34.375101 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:34.417029 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:34.416924 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" podStartSLOduration=2.095522721 podStartE2EDuration="2.416908179s" totalImagesPullingTime="321.385458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:49:34.416531952 +0000 UTC m=+3340.925551858" watchObservedRunningTime="2026-10-30 06:49:34.416908179 +0000 UTC m=+3340.925928067" Oct 30 06:49:34.452336 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:34.452303 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:35.600533 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:35.600486 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:36.070999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:36.070968 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:36.225088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:36.225056 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:36.586276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:36.586240 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:36.975958 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:36.975921 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:38.042156 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:38.042123 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:40.039887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:40.039855 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:40.387643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:40.387616 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:49:40.521142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:40.521111 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:40.667519 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:40.667448 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:42.144268 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:42.144235 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:42.311037 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:42.310996 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:42.452031 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:42.451942 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:44.119197 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:44.119165 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:45.840310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:45.840277 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:47.632083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:47.632050 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:49.460854 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:49.460821 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:51.241161 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:51.241119 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:52.998207 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:52.998176 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:54.902515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:54.902412 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:56.618873 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:56.618839 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:49:58.401237 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:49:58.401170 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:00.217703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:00.217669 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:01.860852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:01.860822 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n"] Oct 30 06:50:01.865693 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:01.865671 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:01.884530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:01.884500 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-89f101-serving-cert\"" Oct 30 06:50:01.904386 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:01.904357 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n"] Oct 30 06:50:01.993845 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:01.993808 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:01.999271 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:01.999252 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pnfx\" (UniqueName: \"kubernetes.io/projected/0f526f75-f591-46d3-8bef-6d9c8b76ded6-kube-api-access-5pnfx\") pod \"maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:01.999347 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:01.999321 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f526f75-f591-46d3-8bef-6d9c8b76ded6-maas-api-tls\") pod \"maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:01.999451 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:01.999431 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f526f75-f591-46d3-8bef-6d9c8b76ded6-metrics-certs\") pod \"maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:01.999541 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:01.999525 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f526f75-f591-46d3-8bef-6d9c8b76ded6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:02.100912 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:02.100877 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f526f75-f591-46d3-8bef-6d9c8b76ded6-maas-api-tls\") pod \"maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:02.102653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:02.101029 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f526f75-f591-46d3-8bef-6d9c8b76ded6-metrics-certs\") pod \"maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:02.102653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:02.101145 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f526f75-f591-46d3-8bef-6d9c8b76ded6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:02.102653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:02.101355 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pnfx\" (UniqueName: \"kubernetes.io/projected/0f526f75-f591-46d3-8bef-6d9c8b76ded6-kube-api-access-5pnfx\") pod \"maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:02.102653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:02.102138 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f526f75-f591-46d3-8bef-6d9c8b76ded6-openshift-service-ca\") pod \"maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:02.104269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:02.104204 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f526f75-f591-46d3-8bef-6d9c8b76ded6-maas-api-tls\") pod \"maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:02.105159 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:02.105139 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f526f75-f591-46d3-8bef-6d9c8b76ded6-metrics-certs\") pod \"maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:02.127862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:02.127782 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pnfx\" (UniqueName: \"kubernetes.io/projected/0f526f75-f591-46d3-8bef-6d9c8b76ded6-kube-api-access-5pnfx\") pod \"maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:02.133358 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:02.133322 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:02.180292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:02.180256 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:02.495018 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:50:02.494986 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f526f75_f591_46d3_8bef_6d9c8b76ded6.slice/crio-584366f766622ac94141746147f5ccbf2a14b159bf906a9a6811ea87cd8bb04f WatchSource:0}: Error finding container 584366f766622ac94141746147f5ccbf2a14b159bf906a9a6811ea87cd8bb04f: Status 404 returned error can't find the container with id 584366f766622ac94141746147f5ccbf2a14b159bf906a9a6811ea87cd8bb04f Oct 30 06:50:02.495180 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:02.495156 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n"] Oct 30 06:50:02.570692 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:02.570649 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" event={"ID":"0f526f75-f591-46d3-8bef-6d9c8b76ded6","Type":"ContainerStarted","Data":"584366f766622ac94141746147f5ccbf2a14b159bf906a9a6811ea87cd8bb04f"} Oct 30 06:50:03.579698 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:03.579665 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" event={"ID":"0f526f75-f591-46d3-8bef-6d9c8b76ded6","Type":"ContainerStarted","Data":"d1fc4fdc53b2ec7436376a4a7a43277f69a4fa8462847a7b4cb380db7af38727"} Oct 30 06:50:03.580201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:03.579828 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:03.622335 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:03.622275 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" podStartSLOduration=2.319335098 podStartE2EDuration="2.622256887s" totalImagesPullingTime="302.921789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:50:03.621423859 +0000 UTC m=+3370.130443758" watchObservedRunningTime="2026-10-30 06:50:03.622256887 +0000 UTC m=+3370.131276774" Oct 30 06:50:03.775573 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:03.775534 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:04.143936 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:50:04.143896 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:50:05.356827 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:05.356790 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:05.647375 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:05.647340 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:07.525889 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:07.525857 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:07.900199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:07.900161 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:09.308353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:09.308300 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:09.592014 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:09.591937 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:50:11.040945 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:11.040916 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:12.804948 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:12.804917 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:14.550077 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:14.550043 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:16.303923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:16.303888 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:18.032445 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:18.032415 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:19.748967 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:19.748936 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:21.551610 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:21.551581 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:23.313335 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:23.313307 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:25.112895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:25.112862 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:26.838889 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:26.838861 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:28.753590 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:28.753557 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:30.645775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:30.645744 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:32.450067 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:32.450023 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:33.169918 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:33.169887 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:33.589314 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:33.589238 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:34.084582 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:34.084548 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:34.243235 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:50:34.239591 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:50:34.244917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:34.244872 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:34.442791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:34.442755 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:34.591644 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:34.591615 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:34.764446 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:34.764341 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:36.218316 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:36.218251 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:38.379871 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:38.379836 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:38.629179 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:38.629144 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:38.862965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:38.862929 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:40.330401 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:40.330373 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:40.470896 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:40.470863 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:42.318761 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:42.318724 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:44.094276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:44.094245 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:45.945958 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:45.945923 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:47.736945 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:47.736914 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:49.434673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:49.434637 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:51.150182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:51.150150 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:53.021643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:53.021609 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:54.754165 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:54.754130 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:56.546762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:56.546731 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:50:58.268137 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:50:58.268074 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:00.033091 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:00.033049 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:01.833359 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:01.833318 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:01.975647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:01.975616 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:03.285692 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.285650 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89"] Oct 30 06:51:03.290777 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.290754 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.314983 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.314943 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89"] Oct 30 06:51:03.322896 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.322872 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-62efb8\"/\"openshift-service-ca.crt\"" Oct 30 06:51:03.323303 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.323272 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-62efb8\"/\"default-dockercfg-mnkjc\"" Oct 30 06:51:03.323917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.323358 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-62efb8\"/\"test-model-62efb8-kserve-self-signed-certs\"" Oct 30 06:51:03.323917 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.323418 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-62efb8\"/\"kube-root-ca.crt\"" Oct 30 06:51:03.371839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.371800 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:03.453820 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.453717 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl28h\" (UniqueName: \"kubernetes.io/projected/1c9f6978-1b17-4e4d-a28b-aed920888f5f-kube-api-access-jl28h\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.454002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.453869 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-tmp-dir\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.454002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.453911 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9f6978-1b17-4e4d-a28b-aed920888f5f-tls-certs\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.454100 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.454075 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-dshm\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.454137 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.454113 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-model-cache\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.454300 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.454282 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-home\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.559460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.556620 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-home\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.559460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.556775 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl28h\" (UniqueName: \"kubernetes.io/projected/1c9f6978-1b17-4e4d-a28b-aed920888f5f-kube-api-access-jl28h\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.559460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.556832 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-tmp-dir\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.559460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.556864 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9f6978-1b17-4e4d-a28b-aed920888f5f-tls-certs\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.559460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.556947 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-dshm\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.559460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.556975 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-model-cache\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.559460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.557542 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-model-cache\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.559460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.557793 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-home\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.559460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.558353 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-tmp-dir\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.568120 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.567882 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9f6978-1b17-4e4d-a28b-aed920888f5f-tls-certs\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.568328 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.568199 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-dshm\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.584244 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.584159 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl28h\" (UniqueName: \"kubernetes.io/projected/1c9f6978-1b17-4e4d-a28b-aed920888f5f-kube-api-access-jl28h\") pod \"test-model-62efb8-kserve-665f8c8757-5fx89\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.615844 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.615509 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:03.829953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.829862 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:03.937204 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:51:03.937172 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c9f6978_1b17_4e4d_a28b_aed920888f5f.slice/crio-b763b363d3ed993a323182d147d7b0547808891732f8b9ae8b15eadcc32918bb WatchSource:0}: Error finding container b763b363d3ed993a323182d147d7b0547808891732f8b9ae8b15eadcc32918bb: Status 404 returned error can't find the container with id b763b363d3ed993a323182d147d7b0547808891732f8b9ae8b15eadcc32918bb Oct 30 06:51:03.939355 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:03.939070 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89"] Oct 30 06:51:04.008609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:04.008575 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" event={"ID":"1c9f6978-1b17-4e4d-a28b-aed920888f5f","Type":"ContainerStarted","Data":"b763b363d3ed993a323182d147d7b0547808891732f8b9ae8b15eadcc32918bb"} Oct 30 06:51:04.145731 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:51:04.145690 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:51:05.016788 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:05.016750 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" event={"ID":"1c9f6978-1b17-4e4d-a28b-aed920888f5f","Type":"ContainerStarted","Data":"0653ca529f2eb563ba821addf554ccb4bc4695571af96e8461afce539e541d82"} Oct 30 06:51:05.061872 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:05.061824 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" podStartSLOduration=2.061810456 podStartE2EDuration="2.061810456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:51:05.058006328 +0000 UTC m=+3431.567026217" watchObservedRunningTime="2026-10-30 06:51:05.061810456 +0000 UTC m=+3431.570830363" Oct 30 06:51:05.521877 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:05.521843 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:05.616131 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:05.616096 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:05.715548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:05.715519 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:07.426143 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:07.426114 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:08.032074 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:08.032045 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:09.140063 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:09.140028 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:11.023569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:11.023537 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:12.825667 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:12.825638 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:13.616356 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:13.616324 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:13.633348 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:13.633320 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:14.093191 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:14.093115 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:51:14.537013 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:14.536982 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:16.337469 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:16.337433 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:18.152259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:18.152181 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:19.924333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:19.924295 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:21.739906 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:21.739869 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:23.520378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:23.520331 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:24.704679 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.704647 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6"] Oct 30 06:51:24.709557 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.709532 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.733353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.733326 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6"] Oct 30 06:51:24.739426 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.739402 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-89f101\"/\"openshift-service-ca.crt\"" Oct 30 06:51:24.739578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.739547 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-89f101\"/\"default-dockercfg-znn2f\"" Oct 30 06:51:24.739578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.739547 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-89f101\"/\"kube-root-ca.crt\"" Oct 30 06:51:24.739704 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.739630 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-89f101\"/\"test-model-89f101-kserve-self-signed-certs\"" Oct 30 06:51:24.783806 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.783766 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:24.840006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.839971 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-model-cache\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.840284 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.840029 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-home\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.840284 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.840253 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-dshm\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.840421 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.840301 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f79fd615-06d2-4a53-85cc-0c7da6834671-tls-certs\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.840529 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.840506 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-tmp-dir\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.840602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.840555 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdbwb\" (UniqueName: \"kubernetes.io/projected/f79fd615-06d2-4a53-85cc-0c7da6834671-kube-api-access-pdbwb\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.948259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.941869 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-dshm\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.948259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.941921 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f79fd615-06d2-4a53-85cc-0c7da6834671-tls-certs\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.948259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.942048 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-tmp-dir\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.948259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.942086 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdbwb\" (UniqueName: \"kubernetes.io/projected/f79fd615-06d2-4a53-85cc-0c7da6834671-kube-api-access-pdbwb\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.948259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.942233 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-model-cache\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.948259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.942268 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-home\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.948259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.942848 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-home\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.948259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.943483 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-tmp-dir\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.948259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.944106 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-model-cache\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.975369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.975256 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdbwb\" (UniqueName: \"kubernetes.io/projected/f79fd615-06d2-4a53-85cc-0c7da6834671-kube-api-access-pdbwb\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.977113 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.977058 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-dshm\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:24.980769 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:24.980722 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f79fd615-06d2-4a53-85cc-0c7da6834671-tls-certs\") pod \"test-model-89f101-kserve-654f586f65-4rjn6\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:25.029628 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:25.029284 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:25.366081 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:25.366053 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6"] Oct 30 06:51:25.366443 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:51:25.366420 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf79fd615_06d2_4a53_85cc_0c7da6834671.slice/crio-78ab806514ad9aede666079a0dbfcd3ee7184f6da84d3f421aa176dfb3b38a01 WatchSource:0}: Error finding container 78ab806514ad9aede666079a0dbfcd3ee7184f6da84d3f421aa176dfb3b38a01: Status 404 returned error can't find the container with id 78ab806514ad9aede666079a0dbfcd3ee7184f6da84d3f421aa176dfb3b38a01 Oct 30 06:51:25.424904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:25.424879 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:26.174659 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:26.174623 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" event={"ID":"f79fd615-06d2-4a53-85cc-0c7da6834671","Type":"ContainerStarted","Data":"c1454e9c62febbcae6f79a3fed2d6288b58561325bafe7b6ea4072814df1d25b"} Oct 30 06:51:26.175057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:26.174675 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" event={"ID":"f79fd615-06d2-4a53-85cc-0c7da6834671","Type":"ContainerStarted","Data":"78ab806514ad9aede666079a0dbfcd3ee7184f6da84d3f421aa176dfb3b38a01"} Oct 30 06:51:26.222026 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:26.221959 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" podStartSLOduration=2.22194511 podStartE2EDuration="2.22194511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:51:26.220052725 +0000 UTC m=+3452.729072617" watchObservedRunningTime="2026-10-30 06:51:26.22194511 +0000 UTC m=+3452.730964998" Oct 30 06:51:27.030625 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:27.030596 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:27.214700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:27.214673 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:29.028690 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:29.028664 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:30.706895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:30.706861 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:32.620457 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:32.620425 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:33.299460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:33.299423 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:34.146545 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:51:34.146507 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:51:34.311107 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.311068 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:34.332965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.332924 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb"] Oct 30 06:51:34.340263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.340238 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:34.357274 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.357199 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-be48b4-serving-cert\"" Oct 30 06:51:34.363290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.363262 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb"] Oct 30 06:51:34.463289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.463163 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:34.475459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.475434 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19ca2c4-c2ed-4d38-9537-fc394357aa84-openshift-service-ca\") pod \"maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:34.475632 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.475615 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19ca2c4-c2ed-4d38-9537-fc394357aa84-maas-api-tls\") pod \"maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:34.475682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.475663 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkswd\" (UniqueName: \"kubernetes.io/projected/a19ca2c4-c2ed-4d38-9537-fc394357aa84-kube-api-access-qkswd\") pod \"maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:34.475801 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.475784 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a19ca2c4-c2ed-4d38-9537-fc394357aa84-metrics-certs\") pod \"maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:34.577031 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.576989 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19ca2c4-c2ed-4d38-9537-fc394357aa84-openshift-service-ca\") pod \"maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:34.577247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.577202 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19ca2c4-c2ed-4d38-9537-fc394357aa84-maas-api-tls\") pod \"maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:34.577342 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.577291 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkswd\" (UniqueName: \"kubernetes.io/projected/a19ca2c4-c2ed-4d38-9537-fc394357aa84-kube-api-access-qkswd\") pod \"maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:34.577403 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.577368 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a19ca2c4-c2ed-4d38-9537-fc394357aa84-metrics-certs\") pod \"maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:34.577857 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.577826 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19ca2c4-c2ed-4d38-9537-fc394357aa84-openshift-service-ca\") pod \"maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:34.585394 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.584125 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19ca2c4-c2ed-4d38-9537-fc394357aa84-maas-api-tls\") pod \"maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:34.589966 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.589942 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a19ca2c4-c2ed-4d38-9537-fc394357aa84-metrics-certs\") pod \"maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:34.604330 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.604253 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkswd\" (UniqueName: \"kubernetes.io/projected/a19ca2c4-c2ed-4d38-9537-fc394357aa84-kube-api-access-qkswd\") pod \"maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:34.662491 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.662435 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:34.678866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.678836 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:34.989739 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:51:34.989669 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda19ca2c4_c2ed_4d38_9537_fc394357aa84.slice/crio-9b42d057d17a3451798b51a8ca7d653e6e658bb9c0c3eca3e87f35195afac434 WatchSource:0}: Error finding container 9b42d057d17a3451798b51a8ca7d653e6e658bb9c0c3eca3e87f35195afac434: Status 404 returned error can't find the container with id 9b42d057d17a3451798b51a8ca7d653e6e658bb9c0c3eca3e87f35195afac434 Oct 30 06:51:34.989888 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:34.989864 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb"] Oct 30 06:51:35.030071 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:35.030039 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:35.042824 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:35.042801 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:35.242747 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:35.242636 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" event={"ID":"a19ca2c4-c2ed-4d38-9537-fc394357aa84","Type":"ContainerStarted","Data":"9b42d057d17a3451798b51a8ca7d653e6e658bb9c0c3eca3e87f35195afac434"} Oct 30 06:51:35.254313 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:35.254292 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:51:36.046567 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:36.046527 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:36.185430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:36.185396 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:36.252002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:36.251969 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" event={"ID":"a19ca2c4-c2ed-4d38-9537-fc394357aa84","Type":"ContainerStarted","Data":"62a393d0e22dae902c633b527f8f18928de16bf1a2587e7be158419d612a1889"} Oct 30 06:51:36.252462 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:36.252056 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:36.295855 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:36.295800 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" podStartSLOduration=1.82089482 podStartE2EDuration="2.295780455s" totalImagesPullingTime="474.885635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:51:36.294304279 +0000 UTC m=+3462.803324167" watchObservedRunningTime="2026-10-30 06:51:36.295780455 +0000 UTC m=+3462.804800345" Oct 30 06:51:36.332233 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:36.332152 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:38.318673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:38.318648 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:40.040526 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:40.040495 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:41.828425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:41.828391 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:42.265603 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:42.265576 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:51:43.620305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:43.620279 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:45.434968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:45.434931 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:47.236390 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:47.236358 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:49.022122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:49.022078 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:50.818017 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:50.817984 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:52.648368 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:52.648317 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:54.437718 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:54.437685 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:56.236077 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:56.236021 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:58.049359 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.049306 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:58.052056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.052027 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf"] Oct 30 06:51:58.059250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.057479 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.085150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.085069 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a5f459\"/\"default-dockercfg-42vbj\"" Oct 30 06:51:58.085150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.085126 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a5f459\"/\"test-model-a5f459-kserve-self-signed-certs\"" Oct 30 06:51:58.085150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.085137 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a5f459\"/\"openshift-service-ca.crt\"" Oct 30 06:51:58.086438 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.086417 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a5f459\"/\"kube-root-ca.crt\"" Oct 30 06:51:58.089133 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.089111 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf"] Oct 30 06:51:58.193240 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.193194 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:51:58.216490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.216457 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-tmp-dir\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.216490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.216494 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxmq7\" (UniqueName: \"kubernetes.io/projected/afadd843-8c76-4a9c-b193-90b1ff6560e7-kube-api-access-lxmq7\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.216757 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.216612 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-dshm\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.216819 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.216735 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-model-cache\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.216883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.216835 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afadd843-8c76-4a9c-b193-90b1ff6560e7-tls-certs\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.216940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.216922 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-home\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.319290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.318189 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afadd843-8c76-4a9c-b193-90b1ff6560e7-tls-certs\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.319290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.318308 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-home\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.319290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.318495 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-tmp-dir\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.319290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.318520 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxmq7\" (UniqueName: \"kubernetes.io/projected/afadd843-8c76-4a9c-b193-90b1ff6560e7-kube-api-access-lxmq7\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.319290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.318585 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-dshm\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.319290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.318673 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-model-cache\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.319290 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.319096 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-model-cache\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.319843 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.319760 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-tmp-dir\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.319843 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.319822 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-home\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.340121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.339874 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afadd843-8c76-4a9c-b193-90b1ff6560e7-tls-certs\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.340121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.340071 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-dshm\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.343958 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.343928 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxmq7\" (UniqueName: \"kubernetes.io/projected/afadd843-8c76-4a9c-b193-90b1ff6560e7-kube-api-access-lxmq7\") pod \"test-model-a5f459-kserve-68d687fc8c-g8gmf\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.383739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.383710 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:51:58.705023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:58.704993 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf"] Oct 30 06:51:58.705506 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:51:58.705480 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafadd843_8c76_4a9c_b193_90b1ff6560e7.slice/crio-51497b3d46e7fde5d1ecfafc0b1975dee739f8d3059178ff445c2b6388d55893 WatchSource:0}: Error finding container 51497b3d46e7fde5d1ecfafc0b1975dee739f8d3059178ff445c2b6388d55893: Status 404 returned error can't find the container with id 51497b3d46e7fde5d1ecfafc0b1975dee739f8d3059178ff445c2b6388d55893 Oct 30 06:51:59.420410 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:59.420376 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" event={"ID":"afadd843-8c76-4a9c-b193-90b1ff6560e7","Type":"ContainerStarted","Data":"aaaa6efe9dd7c9bf7fee63662b72bf705ce8aa6d56a215538ebc3ab656613b8d"} Oct 30 06:51:59.420797 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:59.420421 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" event={"ID":"afadd843-8c76-4a9c-b193-90b1ff6560e7","Type":"ContainerStarted","Data":"51497b3d46e7fde5d1ecfafc0b1975dee739f8d3059178ff445c2b6388d55893"} Oct 30 06:51:59.463910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:51:59.463862 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" podStartSLOduration=1.463848713 podStartE2EDuration="1.463848713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:51:59.461039514 +0000 UTC m=+3485.970059402" watchObservedRunningTime="2026-10-30 06:51:59.463848713 +0000 UTC m=+3485.972868600" Oct 30 06:52:00.035065 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:00.035029 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:00.384630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:00.384601 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:52:01.784409 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:01.784383 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:03.563049 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:03.563015 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:04.144007 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:52:04.143970 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:52:04.992914 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:04.992879 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:05.135604 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:05.135567 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:05.482749 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:05.482707 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:05.620039 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:05.620010 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:05.768404 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:05.768320 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:05.915372 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:05.915342 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:07.342396 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:07.342367 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:08.151101 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:08.151070 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:08.384362 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:08.384334 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:52:08.396938 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:08.396911 6999 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:52:08.499714 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:08.499640 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:52:09.044031 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:09.043997 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:10.830306 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:10.830273 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:12.582689 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:12.582654 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:12.722723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:12.722693 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:14.465894 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:14.465861 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:16.251329 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:16.251295 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:18.036586 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:18.036547 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:19.765736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:19.765702 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:21.543994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:21.543965 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:23.264034 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:23.264000 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:23.500531 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:23.500500 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:23.644413 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:23.644381 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:25.061665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:25.061630 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:26.681118 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:26.681086 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:26.824750 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:26.824714 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:28.582034 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:28.582001 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:30.436168 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:30.436134 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:32.275562 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:32.275527 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:34.024529 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:34.024448 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:34.146107 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:52:34.146065 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:52:35.844318 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:35.844284 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:36.444507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:36.444478 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:37.533727 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:37.533695 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:37.735837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:37.735787 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:39.529773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:39.529742 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:41.288448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:41.288415 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:43.009201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:43.009163 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:44.842704 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:44.842668 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:46.592458 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:46.592421 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:48.459765 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:48.459733 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:50.453028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:50.452993 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:52.413334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:52.413300 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:54.165049 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:54.165009 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:55.954502 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:55.954466 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:57.740708 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:57.740678 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:52:59.536357 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:52:59.536322 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:01.423164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:01.423118 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:02.816807 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:02.816769 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn"] Oct 30 06:53:02.817337 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:02.817022 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" podUID="f523aacd-5b3e-4436-a370-c885f63b23cc" containerName="maas-api" containerID="cri-o://f5933cf2d33aabfa70792d57e999686309c7e359e2908f43420093306c486f22" gracePeriod=30 Oct 30 06:53:02.917200 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:02.917160 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:03.183928 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.183558 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:53:03.233247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.231639 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f523aacd-5b3e-4436-a370-c885f63b23cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f523aacd-5b3e-4436-a370-c885f63b23cc-maas-api-tls\") pod \"f523aacd-5b3e-4436-a370-c885f63b23cc\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " Oct 30 06:53:03.233247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.231718 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f523aacd-5b3e-4436-a370-c885f63b23cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f523aacd-5b3e-4436-a370-c885f63b23cc-metrics-certs\") pod \"f523aacd-5b3e-4436-a370-c885f63b23cc\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " Oct 30 06:53:03.233247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.231850 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f523aacd-5b3e-4436-a370-c885f63b23cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f523aacd-5b3e-4436-a370-c885f63b23cc-openshift-service-ca\") pod \"f523aacd-5b3e-4436-a370-c885f63b23cc\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " Oct 30 06:53:03.233247 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.232075 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f523aacd-5b3e-4436-a370-c885f63b23cc-kube-api-access-ndfk4\" (UniqueName: \"kubernetes.io/projected/f523aacd-5b3e-4436-a370-c885f63b23cc-kube-api-access-ndfk4\") pod \"f523aacd-5b3e-4436-a370-c885f63b23cc\" (UID: \"f523aacd-5b3e-4436-a370-c885f63b23cc\") " Oct 30 06:53:03.237110 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.237074 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f523aacd-5b3e-4436-a370-c885f63b23cc-openshift-service-ca" pod "f523aacd-5b3e-4436-a370-c885f63b23cc" (UID: "f523aacd-5b3e-4436-a370-c885f63b23cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:03.250139 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.250102 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f523aacd-5b3e-4436-a370-c885f63b23cc-kube-api-access-ndfk4" pod "f523aacd-5b3e-4436-a370-c885f63b23cc" (UID: "f523aacd-5b3e-4436-a370-c885f63b23cc"). InnerVolumeSpecName "kube-api-access-ndfk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:03.250975 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.250942 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f523aacd-5b3e-4436-a370-c885f63b23cc-metrics-certs" pod "f523aacd-5b3e-4436-a370-c885f63b23cc" (UID: "f523aacd-5b3e-4436-a370-c885f63b23cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:53:03.251162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.251126 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f523aacd-5b3e-4436-a370-c885f63b23cc-maas-api-tls" pod "f523aacd-5b3e-4436-a370-c885f63b23cc" (UID: "f523aacd-5b3e-4436-a370-c885f63b23cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:53:03.345607 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.335448 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndfk4\" (UniqueName: \"kubernetes.io/projected/f523aacd-5b3e-4436-a370-c885f63b23cc-kube-api-access-ndfk4\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:03.350974 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.350942 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f523aacd-5b3e-4436-a370-c885f63b23cc-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:03.351227 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.351198 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f523aacd-5b3e-4436-a370-c885f63b23cc-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:03.360459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.360433 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f523aacd-5b3e-4436-a370-c885f63b23cc-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:03.586111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.586025 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:03.733375 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.733341 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:03.868582 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.868545 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:03.920448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.920413 6999 generic.go:356] "Generic (PLEG): container finished" podID="f523aacd-5b3e-4436-a370-c885f63b23cc" containerID="f5933cf2d33aabfa70792d57e999686309c7e359e2908f43420093306c486f22" exitCode=0 Oct 30 06:53:03.920658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.920468 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" Oct 30 06:53:03.920658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.920500 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" event={"ID":"f523aacd-5b3e-4436-a370-c885f63b23cc","Type":"ContainerDied","Data":"f5933cf2d33aabfa70792d57e999686309c7e359e2908f43420093306c486f22"} Oct 30 06:53:03.920658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.920561 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn" event={"ID":"f523aacd-5b3e-4436-a370-c885f63b23cc","Type":"ContainerDied","Data":"996fe5972bbb3f80cc0344f24d056396d6a0621a963146ac6dd544c1e4cb3ca3"} Oct 30 06:53:03.920658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.920599 6999 scope.go:122] "RemoveContainer" containerID="f5933cf2d33aabfa70792d57e999686309c7e359e2908f43420093306c486f22" Oct 30 06:53:03.938362 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.938238 6999 scope.go:122] "RemoveContainer" containerID="f5933cf2d33aabfa70792d57e999686309c7e359e2908f43420093306c486f22" Oct 30 06:53:03.946577 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:53:03.946533 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5933cf2d33aabfa70792d57e999686309c7e359e2908f43420093306c486f22\": container with ID starting with f5933cf2d33aabfa70792d57e999686309c7e359e2908f43420093306c486f22 not found: ID does not exist" containerID="f5933cf2d33aabfa70792d57e999686309c7e359e2908f43420093306c486f22" Oct 30 06:53:03.946745 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.946581 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5933cf2d33aabfa70792d57e999686309c7e359e2908f43420093306c486f22"} err="failed to get container status \"f5933cf2d33aabfa70792d57e999686309c7e359e2908f43420093306c486f22\": rpc error: code = NotFound desc = could not find container \"f5933cf2d33aabfa70792d57e999686309c7e359e2908f43420093306c486f22\": container with ID starting with f5933cf2d33aabfa70792d57e999686309c7e359e2908f43420093306c486f22 not found: ID does not exist" Oct 30 06:53:03.977670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.977631 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn"] Oct 30 06:53:03.989269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:03.989240 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2d2923-5c67975fc5-vztdn"] Oct 30 06:53:04.009575 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:04.009528 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:04.147051 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:53:04.146959 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:53:04.256991 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:04.256955 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:05.612523 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:05.612496 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f523aacd-5b3e-4436-a370-c885f63b23cc" path="/var/lib/kubelet/pods/f523aacd-5b3e-4436-a370-c885f63b23cc/volumes" Oct 30 06:53:05.620668 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:05.620644 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:05.920500 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:05.920420 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:07.436253 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:07.436194 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:07.949104 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:07.949062 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw"] Oct 30 06:53:08.017528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:08.017495 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:08.407711 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:08.407681 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:09.290929 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:09.290896 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:10.930196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:10.930166 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:11.076757 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:11.076725 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:11.289165 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:11.289090 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:12.905155 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:12.905123 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:14.798383 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:14.798349 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:15.306824 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.306798 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:15.424853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.424821 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj"] Oct 30 06:53:15.425585 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.425570 6999 cpu_manager.go:414] "containerMap: removing container" podUID="f523aacd-5b3e-4436-a370-c885f63b23cc" containerName="maas-api" Oct 30 06:53:15.425654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.425586 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f523aacd-5b3e-4436-a370-c885f63b23cc" containerName="maas-api" Oct 30 06:53:15.425737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.425728 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="f523aacd-5b3e-4436-a370-c885f63b23cc" containerName="maas-api" Oct 30 06:53:15.430857 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.430840 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj" Oct 30 06:53:15.447772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.447747 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 06:53:15.449535 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.449512 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj"] Oct 30 06:53:15.457144 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.457079 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 06:53:15.457277 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.457149 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 06:53:15.459738 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.459717 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jh7qr\"" Oct 30 06:53:15.459856 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.459806 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 06:53:15.461267 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.461208 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:15.531111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.531088 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/013f3068-30a0-4b4c-b313-6eee3520eb14-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-bwccj\" (UID: \"013f3068-30a0-4b4c-b313-6eee3520eb14\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj" Oct 30 06:53:15.531292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.531264 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/013f3068-30a0-4b4c-b313-6eee3520eb14-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-bwccj\" (UID: \"013f3068-30a0-4b4c-b313-6eee3520eb14\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj" Oct 30 06:53:15.531360 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.531305 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krxgf\" (UniqueName: \"kubernetes.io/projected/013f3068-30a0-4b4c-b313-6eee3520eb14-kube-api-access-krxgf\") pod \"kuadrant-console-plugin-66c4fb575d-bwccj\" (UID: \"013f3068-30a0-4b4c-b313-6eee3520eb14\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj" Oct 30 06:53:15.633327 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.633294 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/013f3068-30a0-4b4c-b313-6eee3520eb14-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-bwccj\" (UID: \"013f3068-30a0-4b4c-b313-6eee3520eb14\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj" Oct 30 06:53:15.633494 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.633387 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/013f3068-30a0-4b4c-b313-6eee3520eb14-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-bwccj\" (UID: \"013f3068-30a0-4b4c-b313-6eee3520eb14\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj" Oct 30 06:53:15.633494 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.633411 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krxgf\" (UniqueName: \"kubernetes.io/projected/013f3068-30a0-4b4c-b313-6eee3520eb14-kube-api-access-krxgf\") pod \"kuadrant-console-plugin-66c4fb575d-bwccj\" (UID: \"013f3068-30a0-4b4c-b313-6eee3520eb14\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj" Oct 30 06:53:15.634059 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.634032 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/013f3068-30a0-4b4c-b313-6eee3520eb14-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-bwccj\" (UID: \"013f3068-30a0-4b4c-b313-6eee3520eb14\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj" Oct 30 06:53:15.635922 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.635901 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/013f3068-30a0-4b4c-b313-6eee3520eb14-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-bwccj\" (UID: \"013f3068-30a0-4b4c-b313-6eee3520eb14\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj" Oct 30 06:53:15.656954 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.656926 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krxgf\" (UniqueName: \"kubernetes.io/projected/013f3068-30a0-4b4c-b313-6eee3520eb14-kube-api-access-krxgf\") pod \"kuadrant-console-plugin-66c4fb575d-bwccj\" (UID: \"013f3068-30a0-4b4c-b313-6eee3520eb14\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj" Oct 30 06:53:15.745131 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:15.745107 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj" Oct 30 06:53:16.057468 ip-10-0-131-226 kubenswrapper[6999]: W1030 06:53:16.057426 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod013f3068_30a0_4b4c_b313_6eee3520eb14.slice/crio-a188c2dc9ea9abe7c747a97b1ac2390653d120d753f5ccc989c5786591bc5f89 WatchSource:0}: Error finding container a188c2dc9ea9abe7c747a97b1ac2390653d120d753f5ccc989c5786591bc5f89: Status 404 returned error can't find the container with id a188c2dc9ea9abe7c747a97b1ac2390653d120d753f5ccc989c5786591bc5f89 Oct 30 06:53:16.058145 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:16.058124 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj"] Oct 30 06:53:16.058876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:16.058859 6999 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:53:16.675171 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:16.675130 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:17.022399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:17.022287 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj" event={"ID":"013f3068-30a0-4b4c-b313-6eee3520eb14","Type":"ContainerStarted","Data":"a188c2dc9ea9abe7c747a97b1ac2390653d120d753f5ccc989c5786591bc5f89"} Oct 30 06:53:18.421851 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:18.421613 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:20.219735 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:20.219696 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:20.665968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:20.665932 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:22.066841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:22.066806 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj" event={"ID":"013f3068-30a0-4b4c-b313-6eee3520eb14","Type":"ContainerStarted","Data":"73c0d2ea692a4931cbb22b8a83e85959847c2c80d39f0d268712349119298fcd"} Oct 30 06:53:22.098741 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:22.098713 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:22.106530 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:22.106479 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-bwccj" podStartSLOduration=1.919811717 podStartE2EDuration="7.106461346s" totalImagesPullingTime="5.186649629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:53:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:53:22.105662391 +0000 UTC m=+3568.614682278" watchObservedRunningTime="2026-10-30 06:53:22.106461346 +0000 UTC m=+3568.615481236" Oct 30 06:53:22.200976 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:22.200946 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp"] Oct 30 06:53:22.237023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:22.236988 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9"] Oct 30 06:53:22.371378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:22.371351 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:22.520096 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:22.520065 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:22.976411 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:22.976375 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" podUID="784f9ba2-3d6b-4cf4-8546-32c975d271ed" containerName="main" containerID="cri-o://dccc539411176d80990278b1b7176980fa3bb5cb91f65aa2adf5d179f1c129ab" gracePeriod=45 Oct 30 06:53:23.089799 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.089731 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" podUID="784f9ba2-3d6b-4cf4-8546-32c975d271ed" containerName="main" probeResult="failure" output="Get \"https://10.134.0.197:8000/ready\": dial tcp 10.134.0.197:8000: connect: connection refused" Oct 30 06:53:23.261569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.261544 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:53:23.382288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.382255 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-home\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-home\") pod \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " Oct 30 06:53:23.382490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.382341 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/784f9ba2-3d6b-4cf4-8546-32c975d271ed-kube-api-access-dwhkf\" (UniqueName: \"kubernetes.io/projected/784f9ba2-3d6b-4cf4-8546-32c975d271ed-kube-api-access-dwhkf\") pod \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " Oct 30 06:53:23.382490 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.382403 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-dshm\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-dshm\") pod \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " Oct 30 06:53:23.382643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.382517 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-tmp-dir\") pod \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " Oct 30 06:53:23.382643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.382570 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-model-cache\") pod \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " Oct 30 06:53:23.382643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.382623 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/784f9ba2-3d6b-4cf4-8546-32c975d271ed-tls-certs\" (UniqueName: \"kubernetes.io/secret/784f9ba2-3d6b-4cf4-8546-32c975d271ed-tls-certs\") pod \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\" (UID: \"784f9ba2-3d6b-4cf4-8546-32c975d271ed\") " Oct 30 06:53:23.382822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.382733 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-home" pod "784f9ba2-3d6b-4cf4-8546-32c975d271ed" (UID: "784f9ba2-3d6b-4cf4-8546-32c975d271ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:23.382957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.382934 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-model-cache" pod "784f9ba2-3d6b-4cf4-8546-32c975d271ed" (UID: "784f9ba2-3d6b-4cf4-8546-32c975d271ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:23.383148 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.382973 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-tmp-dir" pod "784f9ba2-3d6b-4cf4-8546-32c975d271ed" (UID: "784f9ba2-3d6b-4cf4-8546-32c975d271ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:23.383330 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.383307 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:23.383585 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.383337 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:23.383585 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.383352 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:23.385286 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.385258 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-dshm" pod "784f9ba2-3d6b-4cf4-8546-32c975d271ed" (UID: "784f9ba2-3d6b-4cf4-8546-32c975d271ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:23.385286 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.385276 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784f9ba2-3d6b-4cf4-8546-32c975d271ed-tls-certs" pod "784f9ba2-3d6b-4cf4-8546-32c975d271ed" (UID: "784f9ba2-3d6b-4cf4-8546-32c975d271ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:53:23.385430 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.385358 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784f9ba2-3d6b-4cf4-8546-32c975d271ed-kube-api-access-dwhkf" pod "784f9ba2-3d6b-4cf4-8546-32c975d271ed" (UID: "784f9ba2-3d6b-4cf4-8546-32c975d271ed"). InnerVolumeSpecName "kube-api-access-dwhkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:23.485119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.485062 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwhkf\" (UniqueName: \"kubernetes.io/projected/784f9ba2-3d6b-4cf4-8546-32c975d271ed-kube-api-access-dwhkf\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:23.485119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.485086 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/784f9ba2-3d6b-4cf4-8546-32c975d271ed-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:23.485119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.485097 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/784f9ba2-3d6b-4cf4-8546-32c975d271ed-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:23.729713 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.729684 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:23.869468 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:23.869436 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:24.031431 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:24.031399 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:24.088233 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:24.088182 6999 generic.go:356] "Generic (PLEG): container finished" podID="784f9ba2-3d6b-4cf4-8546-32c975d271ed" containerID="dccc539411176d80990278b1b7176980fa3bb5cb91f65aa2adf5d179f1c129ab" exitCode=0 Oct 30 06:53:24.088436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:24.088414 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" event={"ID":"784f9ba2-3d6b-4cf4-8546-32c975d271ed","Type":"ContainerDied","Data":"dccc539411176d80990278b1b7176980fa3bb5cb91f65aa2adf5d179f1c129ab"} Oct 30 06:53:24.088527 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:24.088458 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" event={"ID":"784f9ba2-3d6b-4cf4-8546-32c975d271ed","Type":"ContainerDied","Data":"5a653716665a76dc318c001600bf0160db5f865ae8ba85dcc21b299ae0bc1f57"} Oct 30 06:53:24.088527 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:24.088485 6999 scope.go:122] "RemoveContainer" containerID="dccc539411176d80990278b1b7176980fa3bb5cb91f65aa2adf5d179f1c129ab" Oct 30 06:53:24.088653 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:24.088516 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw" Oct 30 06:53:24.100115 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:24.100092 6999 scope.go:122] "RemoveContainer" containerID="dccc539411176d80990278b1b7176980fa3bb5cb91f65aa2adf5d179f1c129ab" Oct 30 06:53:24.100434 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:53:24.100393 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dccc539411176d80990278b1b7176980fa3bb5cb91f65aa2adf5d179f1c129ab\": container with ID starting with dccc539411176d80990278b1b7176980fa3bb5cb91f65aa2adf5d179f1c129ab not found: ID does not exist" containerID="dccc539411176d80990278b1b7176980fa3bb5cb91f65aa2adf5d179f1c129ab" Oct 30 06:53:24.100434 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:24.100416 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dccc539411176d80990278b1b7176980fa3bb5cb91f65aa2adf5d179f1c129ab"} err="failed to get container status \"dccc539411176d80990278b1b7176980fa3bb5cb91f65aa2adf5d179f1c129ab\": rpc error: code = NotFound desc = could not find container \"dccc539411176d80990278b1b7176980fa3bb5cb91f65aa2adf5d179f1c129ab\": container with ID starting with dccc539411176d80990278b1b7176980fa3bb5cb91f65aa2adf5d179f1c129ab not found: ID does not exist" Oct 30 06:53:24.153563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:24.153509 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw"] Oct 30 06:53:24.165935 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:24.165915 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1c148a/test-model-1c148a-kserve-5d777bdb99-w2cfw"] Oct 30 06:53:25.610437 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:25.610407 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="784f9ba2-3d6b-4cf4-8546-32c975d271ed" path="/var/lib/kubelet/pods/784f9ba2-3d6b-4cf4-8546-32c975d271ed/volumes" Oct 30 06:53:25.808472 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:25.808440 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:27.201243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:27.201168 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" podUID="e4de3e31-9dff-43fc-8553-f859e41a1583" containerName="payload-processing" containerID="cri-o://ad42af138fcb0b7c94699ac5547e4d96395ed3755bc0b9247961cc8d284a31ff" gracePeriod=25 Oct 30 06:53:27.237511 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:27.237444 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" podUID="fb73c18a-0b2c-44b4-a5de-133963b43a88" containerName="payload-pre-processing" containerID="cri-o://fa9e1114aec9b183c703ace40c88f72bc5f4f106aad545e4d8791f9543177c02" gracePeriod=25 Oct 30 06:53:27.544750 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:27.544671 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:29.341846 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:29.341819 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:30.811705 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:30.811667 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" podUID="e4de3e31-9dff-43fc-8553-f859e41a1583" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:53:30.812108 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:30.811722 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" podUID="fb73c18a-0b2c-44b4-a5de-133963b43a88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:53:30.867888 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:30.867854 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms"] Oct 30 06:53:30.868165 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:30.868116 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" podUID="115bfbd1-fef2-4d48-a261-a204e607f077" containerName="maas-api" containerID="cri-o://a4aeda6b027952647bfd9c1993b0cfb41f81588a26d959ee2e1f8c3c693ad0b7" gracePeriod=30 Oct 30 06:53:30.956862 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:30.956828 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:31.020515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.020478 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" podUID="115bfbd1-fef2-4d48-a261-a204e607f077" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.174:8443/health\": dial tcp 10.134.0.174:8443: connect: connection refused" Oct 30 06:53:31.182235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.181721 6999 generic.go:356] "Generic (PLEG): container finished" podID="115bfbd1-fef2-4d48-a261-a204e607f077" containerID="a4aeda6b027952647bfd9c1993b0cfb41f81588a26d959ee2e1f8c3c693ad0b7" exitCode=0 Oct 30 06:53:31.182235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.181932 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" event={"ID":"115bfbd1-fef2-4d48-a261-a204e607f077","Type":"ContainerDied","Data":"a4aeda6b027952647bfd9c1993b0cfb41f81588a26d959ee2e1f8c3c693ad0b7"} Oct 30 06:53:31.275532 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.275118 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:53:31.405234 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.404245 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115bfbd1-fef2-4d48-a261-a204e607f077-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/115bfbd1-fef2-4d48-a261-a204e607f077-openshift-service-ca\") pod \"115bfbd1-fef2-4d48-a261-a204e607f077\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " Oct 30 06:53:31.405234 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.404316 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/115bfbd1-fef2-4d48-a261-a204e607f077-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/115bfbd1-fef2-4d48-a261-a204e607f077-maas-api-tls\") pod \"115bfbd1-fef2-4d48-a261-a204e607f077\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " Oct 30 06:53:31.415235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.410919 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115bfbd1-fef2-4d48-a261-a204e607f077-openshift-service-ca" pod "115bfbd1-fef2-4d48-a261-a204e607f077" (UID: "115bfbd1-fef2-4d48-a261-a204e607f077"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:31.415235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.412080 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/115bfbd1-fef2-4d48-a261-a204e607f077-metrics-certs\" (UniqueName: \"kubernetes.io/secret/115bfbd1-fef2-4d48-a261-a204e607f077-metrics-certs\") pod \"115bfbd1-fef2-4d48-a261-a204e607f077\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " Oct 30 06:53:31.415235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.412518 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115bfbd1-fef2-4d48-a261-a204e607f077-kube-api-access-mx7zq\" (UniqueName: \"kubernetes.io/projected/115bfbd1-fef2-4d48-a261-a204e607f077-kube-api-access-mx7zq\") pod \"115bfbd1-fef2-4d48-a261-a204e607f077\" (UID: \"115bfbd1-fef2-4d48-a261-a204e607f077\") " Oct 30 06:53:31.415235 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.413142 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/115bfbd1-fef2-4d48-a261-a204e607f077-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:31.457806 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.457382 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/115bfbd1-fef2-4d48-a261-a204e607f077-maas-api-tls" pod "115bfbd1-fef2-4d48-a261-a204e607f077" (UID: "115bfbd1-fef2-4d48-a261-a204e607f077"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:53:31.459753 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.459721 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/115bfbd1-fef2-4d48-a261-a204e607f077-metrics-certs" pod "115bfbd1-fef2-4d48-a261-a204e607f077" (UID: "115bfbd1-fef2-4d48-a261-a204e607f077"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:53:31.459972 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.459950 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115bfbd1-fef2-4d48-a261-a204e607f077-kube-api-access-mx7zq" pod "115bfbd1-fef2-4d48-a261-a204e607f077" (UID: "115bfbd1-fef2-4d48-a261-a204e607f077"). InnerVolumeSpecName "kube-api-access-mx7zq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:31.491847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.491814 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:31.515228 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.515184 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx7zq\" (UniqueName: \"kubernetes.io/projected/115bfbd1-fef2-4d48-a261-a204e607f077-kube-api-access-mx7zq\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:31.515228 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.515227 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/115bfbd1-fef2-4d48-a261-a204e607f077-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:31.515392 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.515241 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/115bfbd1-fef2-4d48-a261-a204e607f077-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:31.637839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.637814 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:31.784732 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.784634 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:31.928827 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:31.928777 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:32.191023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:32.190986 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" Oct 30 06:53:32.191243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:32.190987 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms" event={"ID":"115bfbd1-fef2-4d48-a261-a204e607f077","Type":"ContainerDied","Data":"649edbae27e9848b7c810f448075514092275eb0f794c10363c6b51bd7e4d997"} Oct 30 06:53:32.191243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:32.191128 6999 scope.go:122] "RemoveContainer" containerID="a4aeda6b027952647bfd9c1993b0cfb41f81588a26d959ee2e1f8c3c693ad0b7" Oct 30 06:53:32.247578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:32.247546 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms"] Oct 30 06:53:32.257943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:32.257913 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a0083e-d6f6df64f-499ms"] Oct 30 06:53:33.179649 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:33.179620 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:33.610080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:33.610057 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115bfbd1-fef2-4d48-a261-a204e607f077" path="/var/lib/kubelet/pods/115bfbd1-fef2-4d48-a261-a204e607f077/volumes" Oct 30 06:53:33.620602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:33.620573 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:34.144529 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:53:34.144496 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:53:35.442068 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:35.442032 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:37.210960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:37.210920 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:38.656018 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:38.655945 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:38.799947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:38.799914 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:38.951447 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:38.951366 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:39.106238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:39.103488 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:39.493287 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:39.493256 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:40.810809 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:40.810764 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" podUID="e4de3e31-9dff-43fc-8553-f859e41a1583" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:53:40.811258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:40.810771 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" podUID="fb73c18a-0b2c-44b4-a5de-133963b43a88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:53:40.925899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:40.925861 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:42.711060 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:42.711022 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:43.988365 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:43.988329 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6"] Oct 30 06:53:43.988956 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:43.988570 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" podUID="dd69357f-342b-4367-96c0-a98fa610742d" containerName="istio-proxy" containerID="cri-o://a49b54f14ee5bf73e1e12e837ae811b86ef5e591681769b5b6cf018bff66f6ae" gracePeriod=30 Oct 30 06:53:44.428306 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:44.428276 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn"] Oct 30 06:53:44.437615 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:44.437585 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm"] Oct 30 06:53:44.449508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:44.449479 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp"] Oct 30 06:53:44.541294 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:44.541263 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:44.684645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:44.684563 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:46.336616 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:46.336588 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:47.539086 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:47.539061 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" Oct 30 06:53:47.554174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:47.554147 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4de3e31-9dff-43fc-8553-f859e41a1583-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4de3e31-9dff-43fc-8553-f859e41a1583-plugins-config-volume\") pod \"e4de3e31-9dff-43fc-8553-f859e41a1583\" (UID: \"e4de3e31-9dff-43fc-8553-f859e41a1583\") " Oct 30 06:53:47.554322 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:47.554265 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4de3e31-9dff-43fc-8553-f859e41a1583-kube-api-access-r8nht\" (UniqueName: \"kubernetes.io/projected/e4de3e31-9dff-43fc-8553-f859e41a1583-kube-api-access-r8nht\") pod \"e4de3e31-9dff-43fc-8553-f859e41a1583\" (UID: \"e4de3e31-9dff-43fc-8553-f859e41a1583\") " Oct 30 06:53:47.554584 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:47.554563 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4de3e31-9dff-43fc-8553-f859e41a1583-plugins-config-volume" pod "e4de3e31-9dff-43fc-8553-f859e41a1583" (UID: "e4de3e31-9dff-43fc-8553-f859e41a1583"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:47.554848 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:47.554830 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4de3e31-9dff-43fc-8553-f859e41a1583-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:47.556499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:47.556473 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4de3e31-9dff-43fc-8553-f859e41a1583-kube-api-access-r8nht" pod "e4de3e31-9dff-43fc-8553-f859e41a1583" (UID: "e4de3e31-9dff-43fc-8553-f859e41a1583"). InnerVolumeSpecName "kube-api-access-r8nht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:47.570190 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:47.570171 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" Oct 30 06:53:47.655324 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:47.655297 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb73c18a-0b2c-44b4-a5de-133963b43a88-kube-api-access-lnl97\" (UniqueName: \"kubernetes.io/projected/fb73c18a-0b2c-44b4-a5de-133963b43a88-kube-api-access-lnl97\") pod \"fb73c18a-0b2c-44b4-a5de-133963b43a88\" (UID: \"fb73c18a-0b2c-44b4-a5de-133963b43a88\") " Oct 30 06:53:47.655487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:47.655428 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb73c18a-0b2c-44b4-a5de-133963b43a88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb73c18a-0b2c-44b4-a5de-133963b43a88-plugins-config-volume\") pod \"fb73c18a-0b2c-44b4-a5de-133963b43a88\" (UID: \"fb73c18a-0b2c-44b4-a5de-133963b43a88\") " Oct 30 06:53:47.656766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:47.655970 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb73c18a-0b2c-44b4-a5de-133963b43a88-plugins-config-volume" pod "fb73c18a-0b2c-44b4-a5de-133963b43a88" (UID: "fb73c18a-0b2c-44b4-a5de-133963b43a88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:47.656766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:47.656122 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8nht\" (UniqueName: \"kubernetes.io/projected/e4de3e31-9dff-43fc-8553-f859e41a1583-kube-api-access-r8nht\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:47.656766 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:47.656139 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb73c18a-0b2c-44b4-a5de-133963b43a88-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:47.658086 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:47.658058 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb73c18a-0b2c-44b4-a5de-133963b43a88-kube-api-access-lnl97" pod "fb73c18a-0b2c-44b4-a5de-133963b43a88" (UID: "fb73c18a-0b2c-44b4-a5de-133963b43a88"). InnerVolumeSpecName "kube-api-access-lnl97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:47.757414 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:47.757351 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnl97\" (UniqueName: \"kubernetes.io/projected/fb73c18a-0b2c-44b4-a5de-133963b43a88-kube-api-access-lnl97\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:48.092782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.092702 6999 patch_prober.go:29] interesting pod/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:53:48.092782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.092767 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" podUID="dd69357f-342b-4367-96c0-a98fa610742d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:53:48.126682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.126658 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:48.303837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.303805 6999 generic.go:356] "Generic (PLEG): container finished" podID="e4de3e31-9dff-43fc-8553-f859e41a1583" containerID="ad42af138fcb0b7c94699ac5547e4d96395ed3755bc0b9247961cc8d284a31ff" exitCode=0 Oct 30 06:53:48.304000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.303846 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" Oct 30 06:53:48.304000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.303884 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" event={"ID":"e4de3e31-9dff-43fc-8553-f859e41a1583","Type":"ContainerDied","Data":"ad42af138fcb0b7c94699ac5547e4d96395ed3755bc0b9247961cc8d284a31ff"} Oct 30 06:53:48.304000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.303932 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp" event={"ID":"e4de3e31-9dff-43fc-8553-f859e41a1583","Type":"ContainerDied","Data":"3ae9c64acaf8c8f31d6b10fb3f4bea43ebe39fdb1f8b1fa4d11a41b11436aef8"} Oct 30 06:53:48.304000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.303956 6999 scope.go:122] "RemoveContainer" containerID="ad42af138fcb0b7c94699ac5547e4d96395ed3755bc0b9247961cc8d284a31ff" Oct 30 06:53:48.305515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.305489 6999 generic.go:356] "Generic (PLEG): container finished" podID="fb73c18a-0b2c-44b4-a5de-133963b43a88" containerID="fa9e1114aec9b183c703ace40c88f72bc5f4f106aad545e4d8791f9543177c02" exitCode=0 Oct 30 06:53:48.305662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.305555 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" Oct 30 06:53:48.305662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.305637 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" event={"ID":"fb73c18a-0b2c-44b4-a5de-133963b43a88","Type":"ContainerDied","Data":"fa9e1114aec9b183c703ace40c88f72bc5f4f106aad545e4d8791f9543177c02"} Oct 30 06:53:48.305796 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.305675 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9" event={"ID":"fb73c18a-0b2c-44b4-a5de-133963b43a88","Type":"ContainerDied","Data":"02cb342309529ffbfd1160559736fa1758872644f29b67e19a1a622cc6b4d714"} Oct 30 06:53:48.314898 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.314883 6999 scope.go:122] "RemoveContainer" containerID="ad42af138fcb0b7c94699ac5547e4d96395ed3755bc0b9247961cc8d284a31ff" Oct 30 06:53:48.315195 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:53:48.315176 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad42af138fcb0b7c94699ac5547e4d96395ed3755bc0b9247961cc8d284a31ff\": container with ID starting with ad42af138fcb0b7c94699ac5547e4d96395ed3755bc0b9247961cc8d284a31ff not found: ID does not exist" containerID="ad42af138fcb0b7c94699ac5547e4d96395ed3755bc0b9247961cc8d284a31ff" Oct 30 06:53:48.315272 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.315203 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad42af138fcb0b7c94699ac5547e4d96395ed3755bc0b9247961cc8d284a31ff"} err="failed to get container status \"ad42af138fcb0b7c94699ac5547e4d96395ed3755bc0b9247961cc8d284a31ff\": rpc error: code = NotFound desc = could not find container \"ad42af138fcb0b7c94699ac5547e4d96395ed3755bc0b9247961cc8d284a31ff\": container with ID starting with ad42af138fcb0b7c94699ac5547e4d96395ed3755bc0b9247961cc8d284a31ff not found: ID does not exist" Oct 30 06:53:48.315272 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.315241 6999 scope.go:122] "RemoveContainer" containerID="fa9e1114aec9b183c703ace40c88f72bc5f4f106aad545e4d8791f9543177c02" Oct 30 06:53:48.325056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.325035 6999 scope.go:122] "RemoveContainer" containerID="fa9e1114aec9b183c703ace40c88f72bc5f4f106aad545e4d8791f9543177c02" Oct 30 06:53:48.325385 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:53:48.325362 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9e1114aec9b183c703ace40c88f72bc5f4f106aad545e4d8791f9543177c02\": container with ID starting with fa9e1114aec9b183c703ace40c88f72bc5f4f106aad545e4d8791f9543177c02 not found: ID does not exist" containerID="fa9e1114aec9b183c703ace40c88f72bc5f4f106aad545e4d8791f9543177c02" Oct 30 06:53:48.325466 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.325397 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9e1114aec9b183c703ace40c88f72bc5f4f106aad545e4d8791f9543177c02"} err="failed to get container status \"fa9e1114aec9b183c703ace40c88f72bc5f4f106aad545e4d8791f9543177c02\": rpc error: code = NotFound desc = could not find container \"fa9e1114aec9b183c703ace40c88f72bc5f4f106aad545e4d8791f9543177c02\": container with ID starting with fa9e1114aec9b183c703ace40c88f72bc5f4f106aad545e4d8791f9543177c02 not found: ID does not exist" Oct 30 06:53:48.356420 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.356337 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp"] Oct 30 06:53:48.367740 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.367720 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-a0083e-d588694d8-kzrwp"] Oct 30 06:53:48.394088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.394065 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9"] Oct 30 06:53:48.398365 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:48.398345 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-a0083e-7688448cbc-tv9f9"] Oct 30 06:53:49.274125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.274102 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:53:49.315996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.315970 6999 generic.go:356] "Generic (PLEG): container finished" podID="dd69357f-342b-4367-96c0-a98fa610742d" containerID="a49b54f14ee5bf73e1e12e837ae811b86ef5e591681769b5b6cf018bff66f6ae" exitCode=0 Oct 30 06:53:49.316142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.316019 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" Oct 30 06:53:49.316142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.316038 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" event={"ID":"dd69357f-342b-4367-96c0-a98fa610742d","Type":"ContainerDied","Data":"a49b54f14ee5bf73e1e12e837ae811b86ef5e591681769b5b6cf018bff66f6ae"} Oct 30 06:53:49.316142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.316091 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6" event={"ID":"dd69357f-342b-4367-96c0-a98fa610742d","Type":"ContainerDied","Data":"e323d23d9ce3355a83b0b2c7bbafc962e1afda04cf1280b8c8ad369bae029d39"} Oct 30 06:53:49.316142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.316119 6999 scope.go:122] "RemoveContainer" containerID="a49b54f14ee5bf73e1e12e837ae811b86ef5e591681769b5b6cf018bff66f6ae" Oct 30 06:53:49.326113 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.326093 6999 scope.go:122] "RemoveContainer" containerID="a49b54f14ee5bf73e1e12e837ae811b86ef5e591681769b5b6cf018bff66f6ae" Oct 30 06:53:49.326403 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:53:49.326381 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49b54f14ee5bf73e1e12e837ae811b86ef5e591681769b5b6cf018bff66f6ae\": container with ID starting with a49b54f14ee5bf73e1e12e837ae811b86ef5e591681769b5b6cf018bff66f6ae not found: ID does not exist" containerID="a49b54f14ee5bf73e1e12e837ae811b86ef5e591681769b5b6cf018bff66f6ae" Oct 30 06:53:49.326467 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.326412 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49b54f14ee5bf73e1e12e837ae811b86ef5e591681769b5b6cf018bff66f6ae"} err="failed to get container status \"a49b54f14ee5bf73e1e12e837ae811b86ef5e591681769b5b6cf018bff66f6ae\": rpc error: code = NotFound desc = could not find container \"a49b54f14ee5bf73e1e12e837ae811b86ef5e591681769b5b6cf018bff66f6ae\": container with ID starting with a49b54f14ee5bf73e1e12e837ae811b86ef5e591681769b5b6cf018bff66f6ae not found: ID does not exist" Oct 30 06:53:49.375754 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.375728 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-workload-certs\") pod \"dd69357f-342b-4367-96c0-a98fa610742d\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " Oct 30 06:53:49.375931 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.375764 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-credential-socket\") pod \"dd69357f-342b-4367-96c0-a98fa610742d\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " Oct 30 06:53:49.375931 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.375805 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-istio-data\") pod \"dd69357f-342b-4367-96c0-a98fa610742d\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " Oct 30 06:53:49.375931 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.375829 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-workload-socket\") pod \"dd69357f-342b-4367-96c0-a98fa610742d\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " Oct 30 06:53:49.375931 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.375864 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd69357f-342b-4367-96c0-a98fa610742d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd69357f-342b-4367-96c0-a98fa610742d-istiod-ca-cert\") pod \"dd69357f-342b-4367-96c0-a98fa610742d\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " Oct 30 06:53:49.375931 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.375890 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-istio-envoy\") pod \"dd69357f-342b-4367-96c0-a98fa610742d\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " Oct 30 06:53:49.376261 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.376049 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd69357f-342b-4367-96c0-a98fa610742d-istio-token\" (UniqueName: \"kubernetes.io/projected/dd69357f-342b-4367-96c0-a98fa610742d-istio-token\") pod \"dd69357f-342b-4367-96c0-a98fa610742d\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " Oct 30 06:53:49.376261 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.376054 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-workload-certs" pod "dd69357f-342b-4367-96c0-a98fa610742d" (UID: "dd69357f-342b-4367-96c0-a98fa610742d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:49.376261 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.376089 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-credential-socket" pod "dd69357f-342b-4367-96c0-a98fa610742d" (UID: "dd69357f-342b-4367-96c0-a98fa610742d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:49.376439 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.376263 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd69357f-342b-4367-96c0-a98fa610742d-kube-api-access-wsr2h\" (UniqueName: \"kubernetes.io/projected/dd69357f-342b-4367-96c0-a98fa610742d-kube-api-access-wsr2h\") pod \"dd69357f-342b-4367-96c0-a98fa610742d\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " Oct 30 06:53:49.376439 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.376351 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-istio-data" pod "dd69357f-342b-4367-96c0-a98fa610742d" (UID: "dd69357f-342b-4367-96c0-a98fa610742d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:49.376439 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.376387 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd69357f-342b-4367-96c0-a98fa610742d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd69357f-342b-4367-96c0-a98fa610742d-istio-podinfo\") pod \"dd69357f-342b-4367-96c0-a98fa610742d\" (UID: \"dd69357f-342b-4367-96c0-a98fa610742d\") " Oct 30 06:53:49.376785 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.376666 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd69357f-342b-4367-96c0-a98fa610742d-istiod-ca-cert" pod "dd69357f-342b-4367-96c0-a98fa610742d" (UID: "dd69357f-342b-4367-96c0-a98fa610742d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:53:49.377164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.376929 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-workload-socket" pod "dd69357f-342b-4367-96c0-a98fa610742d" (UID: "dd69357f-342b-4367-96c0-a98fa610742d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:49.377164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.377083 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:49.377164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.377106 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:49.377164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.377120 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:49.377164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.377132 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:49.377164 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.377144 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd69357f-342b-4367-96c0-a98fa610742d-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:49.379336 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.379310 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-istio-envoy" pod "dd69357f-342b-4367-96c0-a98fa610742d" (UID: "dd69357f-342b-4367-96c0-a98fa610742d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:49.379449 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.379315 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd69357f-342b-4367-96c0-a98fa610742d-istio-token" pod "dd69357f-342b-4367-96c0-a98fa610742d" (UID: "dd69357f-342b-4367-96c0-a98fa610742d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:49.379515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.379447 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd69357f-342b-4367-96c0-a98fa610742d-istio-podinfo" pod "dd69357f-342b-4367-96c0-a98fa610742d" (UID: "dd69357f-342b-4367-96c0-a98fa610742d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:53:49.379747 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.379717 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd69357f-342b-4367-96c0-a98fa610742d-kube-api-access-wsr2h" pod "dd69357f-342b-4367-96c0-a98fa610742d" (UID: "dd69357f-342b-4367-96c0-a98fa610742d"). InnerVolumeSpecName "kube-api-access-wsr2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:49.478473 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.478446 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd69357f-342b-4367-96c0-a98fa610742d-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:49.478473 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.478474 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsr2h\" (UniqueName: \"kubernetes.io/projected/dd69357f-342b-4367-96c0-a98fa610742d-kube-api-access-wsr2h\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:49.478662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.478485 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd69357f-342b-4367-96c0-a98fa610742d-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:49.478662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.478493 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd69357f-342b-4367-96c0-a98fa610742d-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:49.610355 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.610328 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4de3e31-9dff-43fc-8553-f859e41a1583" path="/var/lib/kubelet/pods/e4de3e31-9dff-43fc-8553-f859e41a1583/volumes" Oct 30 06:53:49.610722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.610709 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb73c18a-0b2c-44b4-a5de-133963b43a88" path="/var/lib/kubelet/pods/fb73c18a-0b2c-44b4-a5de-133963b43a88/volumes" Oct 30 06:53:49.659145 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.659119 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6"] Oct 30 06:53:49.671197 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.671175 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-qvpf6"] Oct 30 06:53:49.843534 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:49.843504 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:50.398791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:50.398760 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:50.539283 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:50.539187 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:51.611665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:51.611634 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd69357f-342b-4367-96c0-a98fa610742d" path="/var/lib/kubelet/pods/dd69357f-342b-4367-96c0-a98fa610742d/volumes" Oct 30 06:53:51.648595 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:51.648563 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:53.367873 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:53.367843 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:55.156780 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:55.156752 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:57.033939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:57.033903 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:58.524558 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:58.524526 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:59.062646 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.062616 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:59.212463 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.212416 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:59.410655 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.410625 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:59.454356 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.454314 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" podUID="8e38cac7-3426-497c-8b84-e7552169bb7c" containerName="main" containerID="cri-o://bdb7708463cbed7a80c3e23d25870c524d1633ab0ae4c046887bb8e6aae46fed" gracePeriod=45 Oct 30 06:53:59.463943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.463900 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" podUID="b2701c51-6377-411c-9924-715ff0765564" containerName="main" containerID="cri-o://7f41ba26531816fdba6acf988b99d9b60481a2c5173f25fa42f473cb043b6102" gracePeriod=45 Oct 30 06:53:59.472930 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.472900 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" podUID="776861d0-f966-44ca-b0eb-22b1a1b47e79" containerName="main" containerID="cri-o://484d8bed666746ba0dca30291dc74239b714663a8d0df6943654b6f58390a63d" gracePeriod=45 Oct 30 06:53:59.669583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.669500 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" podUID="b2701c51-6377-411c-9924-715ff0765564" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": dial tcp 10.134.0.55:8000: connect: connection refused" Oct 30 06:53:59.779514 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.779484 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:53:59.785673 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.785644 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:53:59.787641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.787621 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:53:59.789950 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.789925 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:53:59.923777 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.923683 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776861d0-f966-44ca-b0eb-22b1a1b47e79-kube-api-access-9ggk8\" (UniqueName: \"kubernetes.io/projected/776861d0-f966-44ca-b0eb-22b1a1b47e79-kube-api-access-9ggk8\") pod \"776861d0-f966-44ca-b0eb-22b1a1b47e79\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " Oct 30 06:53:59.923777 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.923757 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-dshm\") pod \"b2701c51-6377-411c-9924-715ff0765564\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " Oct 30 06:53:59.924021 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.923806 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-model-cache\") pod \"8e38cac7-3426-497c-8b84-e7552169bb7c\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " Oct 30 06:53:59.924109 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.924089 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-model-cache" pod "8e38cac7-3426-497c-8b84-e7552169bb7c" (UID: "8e38cac7-3426-497c-8b84-e7552169bb7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:59.924471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.924377 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e38cac7-3426-497c-8b84-e7552169bb7c-kube-api-access-w94zl\" (UniqueName: \"kubernetes.io/projected/8e38cac7-3426-497c-8b84-e7552169bb7c-kube-api-access-w94zl\") pod \"8e38cac7-3426-497c-8b84-e7552169bb7c\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " Oct 30 06:53:59.924471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.924421 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-home\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-home\") pod \"8e38cac7-3426-497c-8b84-e7552169bb7c\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " Oct 30 06:53:59.924722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.924482 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-home\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-home\") pod \"b2701c51-6377-411c-9924-715ff0765564\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " Oct 30 06:53:59.924722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.924522 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2701c51-6377-411c-9924-715ff0765564-tls-certs\" (UniqueName: \"kubernetes.io/secret/b2701c51-6377-411c-9924-715ff0765564-tls-certs\") pod \"b2701c51-6377-411c-9924-715ff0765564\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " Oct 30 06:53:59.924722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.924564 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-dshm\") pod \"776861d0-f966-44ca-b0eb-22b1a1b47e79\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " Oct 30 06:53:59.924722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.924595 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-model-cache\") pod \"776861d0-f966-44ca-b0eb-22b1a1b47e79\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " Oct 30 06:53:59.924722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.924681 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e38cac7-3426-497c-8b84-e7552169bb7c-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e38cac7-3426-497c-8b84-e7552169bb7c-tls-certs\") pod \"8e38cac7-3426-497c-8b84-e7552169bb7c\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " Oct 30 06:53:59.925057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.924730 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-home\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-home\") pod \"776861d0-f966-44ca-b0eb-22b1a1b47e79\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " Oct 30 06:53:59.925057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.924774 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-dshm\") pod \"8e38cac7-3426-497c-8b84-e7552169bb7c\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " Oct 30 06:53:59.925057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.924815 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-tmp-dir\") pod \"b2701c51-6377-411c-9924-715ff0765564\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " Oct 30 06:53:59.925057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.924893 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-tmp-dir\") pod \"776861d0-f966-44ca-b0eb-22b1a1b47e79\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " Oct 30 06:53:59.925057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.924918 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2701c51-6377-411c-9924-715ff0765564-kube-api-access-jd2wh\" (UniqueName: \"kubernetes.io/projected/b2701c51-6377-411c-9924-715ff0765564-kube-api-access-jd2wh\") pod \"b2701c51-6377-411c-9924-715ff0765564\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " Oct 30 06:53:59.925057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.924974 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-tmp-dir\") pod \"8e38cac7-3426-497c-8b84-e7552169bb7c\" (UID: \"8e38cac7-3426-497c-8b84-e7552169bb7c\") " Oct 30 06:53:59.925057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.925007 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/776861d0-f966-44ca-b0eb-22b1a1b47e79-tls-certs\" (UniqueName: \"kubernetes.io/secret/776861d0-f966-44ca-b0eb-22b1a1b47e79-tls-certs\") pod \"776861d0-f966-44ca-b0eb-22b1a1b47e79\" (UID: \"776861d0-f966-44ca-b0eb-22b1a1b47e79\") " Oct 30 06:53:59.925057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.925053 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-model-cache\") pod \"b2701c51-6377-411c-9924-715ff0765564\" (UID: \"b2701c51-6377-411c-9924-715ff0765564\") " Oct 30 06:53:59.925524 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.925434 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-home" pod "8e38cac7-3426-497c-8b84-e7552169bb7c" (UID: "8e38cac7-3426-497c-8b84-e7552169bb7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:59.925655 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.925631 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-model-cache" pod "776861d0-f966-44ca-b0eb-22b1a1b47e79" (UID: "776861d0-f966-44ca-b0eb-22b1a1b47e79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:59.926125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.926101 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:59.926192 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.926133 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:59.926192 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.926146 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:53:59.926532 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.926490 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-home" pod "776861d0-f966-44ca-b0eb-22b1a1b47e79" (UID: "776861d0-f966-44ca-b0eb-22b1a1b47e79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:59.930762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.927809 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-model-cache" pod "b2701c51-6377-411c-9924-715ff0765564" (UID: "b2701c51-6377-411c-9924-715ff0765564"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:59.930762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.929831 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776861d0-f966-44ca-b0eb-22b1a1b47e79-kube-api-access-9ggk8" pod "776861d0-f966-44ca-b0eb-22b1a1b47e79" (UID: "776861d0-f966-44ca-b0eb-22b1a1b47e79"). InnerVolumeSpecName "kube-api-access-9ggk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:59.930762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.930036 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-home" pod "b2701c51-6377-411c-9924-715ff0765564" (UID: "b2701c51-6377-411c-9924-715ff0765564"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:59.932731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.932041 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-dshm" pod "776861d0-f966-44ca-b0eb-22b1a1b47e79" (UID: "776861d0-f966-44ca-b0eb-22b1a1b47e79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:59.932731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.932295 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-tmp-dir" pod "b2701c51-6377-411c-9924-715ff0765564" (UID: "b2701c51-6377-411c-9924-715ff0765564"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:59.932731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.932487 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-tmp-dir" pod "8e38cac7-3426-497c-8b84-e7552169bb7c" (UID: "8e38cac7-3426-497c-8b84-e7552169bb7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:59.933701 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.933581 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-dshm" pod "8e38cac7-3426-497c-8b84-e7552169bb7c" (UID: "8e38cac7-3426-497c-8b84-e7552169bb7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:59.933701 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.933661 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e38cac7-3426-497c-8b84-e7552169bb7c-tls-certs" pod "8e38cac7-3426-497c-8b84-e7552169bb7c" (UID: "8e38cac7-3426-497c-8b84-e7552169bb7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:53:59.933894 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.933790 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-tmp-dir" pod "776861d0-f966-44ca-b0eb-22b1a1b47e79" (UID: "776861d0-f966-44ca-b0eb-22b1a1b47e79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:59.934376 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.934345 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2701c51-6377-411c-9924-715ff0765564-kube-api-access-jd2wh" pod "b2701c51-6377-411c-9924-715ff0765564" (UID: "b2701c51-6377-411c-9924-715ff0765564"). InnerVolumeSpecName "kube-api-access-jd2wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:59.938328 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.938301 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776861d0-f966-44ca-b0eb-22b1a1b47e79-tls-certs" pod "776861d0-f966-44ca-b0eb-22b1a1b47e79" (UID: "776861d0-f966-44ca-b0eb-22b1a1b47e79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:53:59.938598 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.938577 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e38cac7-3426-497c-8b84-e7552169bb7c-kube-api-access-w94zl" pod "8e38cac7-3426-497c-8b84-e7552169bb7c" (UID: "8e38cac7-3426-497c-8b84-e7552169bb7c"). InnerVolumeSpecName "kube-api-access-w94zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:53:59.938598 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.938582 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-dshm" pod "b2701c51-6377-411c-9924-715ff0765564" (UID: "b2701c51-6377-411c-9924-715ff0765564"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:53:59.940326 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:53:59.940298 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2701c51-6377-411c-9924-715ff0765564-tls-certs" pod "b2701c51-6377-411c-9924-715ff0765564" (UID: "b2701c51-6377-411c-9924-715ff0765564"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:54:00.027870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.027833 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.027870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.027867 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2701c51-6377-411c-9924-715ff0765564-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.028132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.027881 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.028132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.027892 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e38cac7-3426-497c-8b84-e7552169bb7c-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.028132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.027905 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.028132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.027916 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.028132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.027926 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.028132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.027936 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/776861d0-f966-44ca-b0eb-22b1a1b47e79-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.028132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.027947 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd2wh\" (UniqueName: \"kubernetes.io/projected/b2701c51-6377-411c-9924-715ff0765564-kube-api-access-jd2wh\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.028132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.027959 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e38cac7-3426-497c-8b84-e7552169bb7c-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.028132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.027971 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/776861d0-f966-44ca-b0eb-22b1a1b47e79-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.028132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.027982 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.028132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.027993 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ggk8\" (UniqueName: \"kubernetes.io/projected/776861d0-f966-44ca-b0eb-22b1a1b47e79-kube-api-access-9ggk8\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.028132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.028003 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2701c51-6377-411c-9924-715ff0765564-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.028132 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.028015 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w94zl\" (UniqueName: \"kubernetes.io/projected/8e38cac7-3426-497c-8b84-e7552169bb7c-kube-api-access-w94zl\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:00.057058 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.057025 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg"] Oct 30 06:54:00.057336 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.057306 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" podUID="e88d9911-2196-41cd-a9ae-3e73aca123e5" containerName="istio-proxy" containerID="cri-o://4843e594a62b341adfb645c2ee2ddaa62aec897dcd46339372629a748c652749" gracePeriod=30 Oct 30 06:54:00.401736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.401701 6999 generic.go:356] "Generic (PLEG): container finished" podID="8e38cac7-3426-497c-8b84-e7552169bb7c" containerID="bdb7708463cbed7a80c3e23d25870c524d1633ab0ae4c046887bb8e6aae46fed" exitCode=0 Oct 30 06:54:00.401940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.401744 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" Oct 30 06:54:00.401940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.401788 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" event={"ID":"8e38cac7-3426-497c-8b84-e7552169bb7c","Type":"ContainerDied","Data":"bdb7708463cbed7a80c3e23d25870c524d1633ab0ae4c046887bb8e6aae46fed"} Oct 30 06:54:00.401940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.401830 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn" event={"ID":"8e38cac7-3426-497c-8b84-e7552169bb7c","Type":"ContainerDied","Data":"6ca143acc2df8e646a79e47bd3637351d1d433956641f65a671122ce3eba885b"} Oct 30 06:54:00.401940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.401852 6999 scope.go:122] "RemoveContainer" containerID="bdb7708463cbed7a80c3e23d25870c524d1633ab0ae4c046887bb8e6aae46fed" Oct 30 06:54:00.403343 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.403315 6999 generic.go:356] "Generic (PLEG): container finished" podID="b2701c51-6377-411c-9924-715ff0765564" containerID="7f41ba26531816fdba6acf988b99d9b60481a2c5173f25fa42f473cb043b6102" exitCode=0 Oct 30 06:54:00.403486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.403355 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" Oct 30 06:54:00.403486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.403384 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" event={"ID":"b2701c51-6377-411c-9924-715ff0765564","Type":"ContainerDied","Data":"7f41ba26531816fdba6acf988b99d9b60481a2c5173f25fa42f473cb043b6102"} Oct 30 06:54:00.403486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.403425 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm" event={"ID":"b2701c51-6377-411c-9924-715ff0765564","Type":"ContainerDied","Data":"c19d7e80656cbd1366f01ce47e41760296daf58f197c9dfa44ff38afa7d2f250"} Oct 30 06:54:00.405121 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.405098 6999 generic.go:356] "Generic (PLEG): container finished" podID="776861d0-f966-44ca-b0eb-22b1a1b47e79" containerID="484d8bed666746ba0dca30291dc74239b714663a8d0df6943654b6f58390a63d" exitCode=0 Oct 30 06:54:00.405259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.405145 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" Oct 30 06:54:00.405259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.405161 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" event={"ID":"776861d0-f966-44ca-b0eb-22b1a1b47e79","Type":"ContainerDied","Data":"484d8bed666746ba0dca30291dc74239b714663a8d0df6943654b6f58390a63d"} Oct 30 06:54:00.405259 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.405187 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp" event={"ID":"776861d0-f966-44ca-b0eb-22b1a1b47e79","Type":"ContainerDied","Data":"2a7229960c9d9e03d4af030b11d6a9a61ee3380881cac174849514e06850ee9c"} Oct 30 06:54:00.413647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.413625 6999 scope.go:122] "RemoveContainer" containerID="bdb7708463cbed7a80c3e23d25870c524d1633ab0ae4c046887bb8e6aae46fed" Oct 30 06:54:00.413943 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:54:00.413921 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb7708463cbed7a80c3e23d25870c524d1633ab0ae4c046887bb8e6aae46fed\": container with ID starting with bdb7708463cbed7a80c3e23d25870c524d1633ab0ae4c046887bb8e6aae46fed not found: ID does not exist" containerID="bdb7708463cbed7a80c3e23d25870c524d1633ab0ae4c046887bb8e6aae46fed" Oct 30 06:54:00.414030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.413948 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb7708463cbed7a80c3e23d25870c524d1633ab0ae4c046887bb8e6aae46fed"} err="failed to get container status \"bdb7708463cbed7a80c3e23d25870c524d1633ab0ae4c046887bb8e6aae46fed\": rpc error: code = NotFound desc = could not find container \"bdb7708463cbed7a80c3e23d25870c524d1633ab0ae4c046887bb8e6aae46fed\": container with ID starting with bdb7708463cbed7a80c3e23d25870c524d1633ab0ae4c046887bb8e6aae46fed not found: ID does not exist" Oct 30 06:54:00.414030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.413967 6999 scope.go:122] "RemoveContainer" containerID="7f41ba26531816fdba6acf988b99d9b60481a2c5173f25fa42f473cb043b6102" Oct 30 06:54:00.423193 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.423175 6999 scope.go:122] "RemoveContainer" containerID="7f41ba26531816fdba6acf988b99d9b60481a2c5173f25fa42f473cb043b6102" Oct 30 06:54:00.424157 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:54:00.423463 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f41ba26531816fdba6acf988b99d9b60481a2c5173f25fa42f473cb043b6102\": container with ID starting with 7f41ba26531816fdba6acf988b99d9b60481a2c5173f25fa42f473cb043b6102 not found: ID does not exist" containerID="7f41ba26531816fdba6acf988b99d9b60481a2c5173f25fa42f473cb043b6102" Oct 30 06:54:00.424157 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.423494 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f41ba26531816fdba6acf988b99d9b60481a2c5173f25fa42f473cb043b6102"} err="failed to get container status \"7f41ba26531816fdba6acf988b99d9b60481a2c5173f25fa42f473cb043b6102\": rpc error: code = NotFound desc = could not find container \"7f41ba26531816fdba6acf988b99d9b60481a2c5173f25fa42f473cb043b6102\": container with ID starting with 7f41ba26531816fdba6acf988b99d9b60481a2c5173f25fa42f473cb043b6102 not found: ID does not exist" Oct 30 06:54:00.424157 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.423513 6999 scope.go:122] "RemoveContainer" containerID="484d8bed666746ba0dca30291dc74239b714663a8d0df6943654b6f58390a63d" Oct 30 06:54:00.433299 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.433283 6999 scope.go:122] "RemoveContainer" containerID="484d8bed666746ba0dca30291dc74239b714663a8d0df6943654b6f58390a63d" Oct 30 06:54:00.433547 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:54:00.433528 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484d8bed666746ba0dca30291dc74239b714663a8d0df6943654b6f58390a63d\": container with ID starting with 484d8bed666746ba0dca30291dc74239b714663a8d0df6943654b6f58390a63d not found: ID does not exist" containerID="484d8bed666746ba0dca30291dc74239b714663a8d0df6943654b6f58390a63d" Oct 30 06:54:00.433619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.433552 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484d8bed666746ba0dca30291dc74239b714663a8d0df6943654b6f58390a63d"} err="failed to get container status \"484d8bed666746ba0dca30291dc74239b714663a8d0df6943654b6f58390a63d\": rpc error: code = NotFound desc = could not find container \"484d8bed666746ba0dca30291dc74239b714663a8d0df6943654b6f58390a63d\": container with ID starting with 484d8bed666746ba0dca30291dc74239b714663a8d0df6943654b6f58390a63d not found: ID does not exist" Oct 30 06:54:00.467790 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.467772 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn"] Oct 30 06:54:00.471857 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.471837 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/distinct-w4-eaee88-kserve-7b697c9b57-f58hn"] Oct 30 06:54:00.522581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.522559 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp"] Oct 30 06:54:00.526454 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.526435 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/simulator-w4-eaee88-kserve-67c4785f4c-gn4zp"] Oct 30 06:54:00.560310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.560290 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm"] Oct 30 06:54:00.565845 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:00.565815 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-eaee88/premium-w4-eaee88-kserve-5c444cbcf4-dj7dm"] Oct 30 06:54:01.276394 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:01.276363 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:01.610504 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:01.610479 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="776861d0-f966-44ca-b0eb-22b1a1b47e79" path="/var/lib/kubelet/pods/776861d0-f966-44ca-b0eb-22b1a1b47e79/volumes" Oct 30 06:54:01.610942 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:01.610929 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e38cac7-3426-497c-8b84-e7552169bb7c" path="/var/lib/kubelet/pods/8e38cac7-3426-497c-8b84-e7552169bb7c/volumes" Oct 30 06:54:01.611351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:01.611338 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2701c51-6377-411c-9924-715ff0765564" path="/var/lib/kubelet/pods/b2701c51-6377-411c-9924-715ff0765564/volumes" Oct 30 06:54:03.052821 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:03.052772 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:03.417521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:03.417443 6999 patch_prober.go:29] interesting pod/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:54:03.417521 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:03.417498 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" podUID="e88d9911-2196-41cd-a9ae-3e73aca123e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:54:04.142025 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:54:04.141984 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:54:04.902450 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:04.902416 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:05.329368 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.329346 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:54:05.450662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.450589 6999 generic.go:356] "Generic (PLEG): container finished" podID="e88d9911-2196-41cd-a9ae-3e73aca123e5" containerID="4843e594a62b341adfb645c2ee2ddaa62aec897dcd46339372629a748c652749" exitCode=0 Oct 30 06:54:05.450810 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.450692 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" Oct 30 06:54:05.450810 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.450733 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" event={"ID":"e88d9911-2196-41cd-a9ae-3e73aca123e5","Type":"ContainerDied","Data":"4843e594a62b341adfb645c2ee2ddaa62aec897dcd46339372629a748c652749"} Oct 30 06:54:05.450810 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.450784 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg" event={"ID":"e88d9911-2196-41cd-a9ae-3e73aca123e5","Type":"ContainerDied","Data":"c0bd2873151d5cfc3d6c04c17a4814a9d1fece7537862be35d7cc1c37235b1ef"} Oct 30 06:54:05.450810 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.450804 6999 scope.go:122] "RemoveContainer" containerID="4843e594a62b341adfb645c2ee2ddaa62aec897dcd46339372629a748c652749" Oct 30 06:54:05.460923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.460895 6999 scope.go:122] "RemoveContainer" containerID="4843e594a62b341adfb645c2ee2ddaa62aec897dcd46339372629a748c652749" Oct 30 06:54:05.461199 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:54:05.461178 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4843e594a62b341adfb645c2ee2ddaa62aec897dcd46339372629a748c652749\": container with ID starting with 4843e594a62b341adfb645c2ee2ddaa62aec897dcd46339372629a748c652749 not found: ID does not exist" containerID="4843e594a62b341adfb645c2ee2ddaa62aec897dcd46339372629a748c652749" Oct 30 06:54:05.461314 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.461224 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4843e594a62b341adfb645c2ee2ddaa62aec897dcd46339372629a748c652749"} err="failed to get container status \"4843e594a62b341adfb645c2ee2ddaa62aec897dcd46339372629a748c652749\": rpc error: code = NotFound desc = could not find container \"4843e594a62b341adfb645c2ee2ddaa62aec897dcd46339372629a748c652749\": container with ID starting with 4843e594a62b341adfb645c2ee2ddaa62aec897dcd46339372629a748c652749 not found: ID does not exist" Oct 30 06:54:05.497764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.496987 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88d9911-2196-41cd-a9ae-3e73aca123e5-kube-api-access-7pkcc\" (UniqueName: \"kubernetes.io/projected/e88d9911-2196-41cd-a9ae-3e73aca123e5-kube-api-access-7pkcc\") pod \"e88d9911-2196-41cd-a9ae-3e73aca123e5\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " Oct 30 06:54:05.497764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.497169 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-workload-socket\") pod \"e88d9911-2196-41cd-a9ae-3e73aca123e5\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " Oct 30 06:54:05.497764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.497240 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-podinfo\") pod \"e88d9911-2196-41cd-a9ae-3e73aca123e5\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " Oct 30 06:54:05.497764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.497360 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e88d9911-2196-41cd-a9ae-3e73aca123e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88d9911-2196-41cd-a9ae-3e73aca123e5-istiod-ca-cert\") pod \"e88d9911-2196-41cd-a9ae-3e73aca123e5\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " Oct 30 06:54:05.497764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.497428 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-data\") pod \"e88d9911-2196-41cd-a9ae-3e73aca123e5\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " Oct 30 06:54:05.497764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.497556 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-workload-certs\") pod \"e88d9911-2196-41cd-a9ae-3e73aca123e5\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " Oct 30 06:54:05.497764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.497598 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-credential-socket\") pod \"e88d9911-2196-41cd-a9ae-3e73aca123e5\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " Oct 30 06:54:05.497764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.497659 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-envoy\") pod \"e88d9911-2196-41cd-a9ae-3e73aca123e5\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " Oct 30 06:54:05.497764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.497727 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-workload-socket" pod "e88d9911-2196-41cd-a9ae-3e73aca123e5" (UID: "e88d9911-2196-41cd-a9ae-3e73aca123e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:05.498481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.497944 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-data" pod "e88d9911-2196-41cd-a9ae-3e73aca123e5" (UID: "e88d9911-2196-41cd-a9ae-3e73aca123e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:05.498481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.497953 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-credential-socket" pod "e88d9911-2196-41cd-a9ae-3e73aca123e5" (UID: "e88d9911-2196-41cd-a9ae-3e73aca123e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:05.498481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.498010 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-workload-certs" pod "e88d9911-2196-41cd-a9ae-3e73aca123e5" (UID: "e88d9911-2196-41cd-a9ae-3e73aca123e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:05.498481 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.498137 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88d9911-2196-41cd-a9ae-3e73aca123e5-istiod-ca-cert" pod "e88d9911-2196-41cd-a9ae-3e73aca123e5" (UID: "e88d9911-2196-41cd-a9ae-3e73aca123e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:05.498781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.498759 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-token\" (UniqueName: \"kubernetes.io/projected/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-token\") pod \"e88d9911-2196-41cd-a9ae-3e73aca123e5\" (UID: \"e88d9911-2196-41cd-a9ae-3e73aca123e5\") " Oct 30 06:54:05.499716 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.499550 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:05.499716 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.499578 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:05.499716 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.499595 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88d9911-2196-41cd-a9ae-3e73aca123e5-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:05.499716 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.499607 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:05.499716 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.499619 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:05.500508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.500474 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-envoy" pod "e88d9911-2196-41cd-a9ae-3e73aca123e5" (UID: "e88d9911-2196-41cd-a9ae-3e73aca123e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:54:05.500632 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.500515 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88d9911-2196-41cd-a9ae-3e73aca123e5-kube-api-access-7pkcc" pod "e88d9911-2196-41cd-a9ae-3e73aca123e5" (UID: "e88d9911-2196-41cd-a9ae-3e73aca123e5"). InnerVolumeSpecName "kube-api-access-7pkcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:05.500829 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.500808 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-podinfo" pod "e88d9911-2196-41cd-a9ae-3e73aca123e5" (UID: "e88d9911-2196-41cd-a9ae-3e73aca123e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:54:05.501445 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.501425 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-token" pod "e88d9911-2196-41cd-a9ae-3e73aca123e5" (UID: "e88d9911-2196-41cd-a9ae-3e73aca123e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:05.600416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.600388 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:05.600416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.600413 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:05.600643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.600427 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pkcc\" (UniqueName: \"kubernetes.io/projected/e88d9911-2196-41cd-a9ae-3e73aca123e5-kube-api-access-7pkcc\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:05.600643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.600439 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88d9911-2196-41cd-a9ae-3e73aca123e5-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:05.791977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.791904 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg"] Oct 30 06:54:05.804283 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:05.804253 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8d44dd-openshift-default-7d6bc946f5-t5kbg"] Oct 30 06:54:06.053500 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:06.053421 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:06.337921 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:06.337846 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:06.477134 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:06.477104 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:06.621258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:06.621226 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:06.778241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:06.776017 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:07.609943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:07.609908 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e88d9911-2196-41cd-a9ae-3e73aca123e5" path="/var/lib/kubelet/pods/e88d9911-2196-41cd-a9ae-3e73aca123e5/volumes" Oct 30 06:54:08.456422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:08.456390 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:08.601767 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:08.601741 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:08.737598 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:08.737519 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:10.244054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:10.244018 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:12.040285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:12.040254 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:13.933109 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:13.933074 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:15.711467 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:15.711435 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:17.540569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:17.540539 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:17.900828 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:17.900797 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:18.042225 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:18.042175 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:19.425280 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:19.425248 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:21.165361 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:21.165332 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:22.598850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:22.598810 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:22.743540 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:22.743502 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:23.034565 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:23.034513 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:23.815507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:23.815472 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:24.855792 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:24.855760 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:26.629700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:26.629665 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:28.438252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:28.438199 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:29.876850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:29.876790 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:30.011105 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:30.011072 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:30.153516 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:30.153421 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:31.947638 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:31.947608 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:33.720025 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:33.719978 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:34.145072 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:54:34.145037 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:54:35.455969 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:35.455928 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:37.275053 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:37.275022 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:39.022643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:39.022611 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:40.725631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:40.725604 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:42.637446 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:42.637411 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:44.400622 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:44.400583 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:46.181401 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:46.181371 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:47.939443 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:47.939414 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:49.432844 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:49.432811 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:49.996635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:49.996591 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:50.141773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:50.141740 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:50.634981 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:50.634952 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:51.933980 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:51.933943 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:53.804774 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:53.804743 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:55.269280 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.269237 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw"] Oct 30 06:54:55.269799 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.269473 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" podUID="9cf76c9b-fa0c-4575-aeae-c14e7487809b" containerName="maas-api" containerID="cri-o://981c4edb59f0defddc66ffd1764ce076a41a7d893f50a7123af0846f8fa28aaa" gracePeriod=30 Oct 30 06:54:55.369080 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.369053 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:55.595983 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.595957 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:54:55.629254 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.629205 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-maas-api-tls\") pod \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " Oct 30 06:54:55.629440 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.629302 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cf76c9b-fa0c-4575-aeae-c14e7487809b-kube-api-access-xzj68\" (UniqueName: \"kubernetes.io/projected/9cf76c9b-fa0c-4575-aeae-c14e7487809b-kube-api-access-xzj68\") pod \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " Oct 30 06:54:55.629520 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.629444 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cf76c9b-fa0c-4575-aeae-c14e7487809b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cf76c9b-fa0c-4575-aeae-c14e7487809b-openshift-service-ca\") pod \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " Oct 30 06:54:55.629520 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.629493 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-metrics-certs\") pod \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\" (UID: \"9cf76c9b-fa0c-4575-aeae-c14e7487809b\") " Oct 30 06:54:55.629934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.629899 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cf76c9b-fa0c-4575-aeae-c14e7487809b-openshift-service-ca" pod "9cf76c9b-fa0c-4575-aeae-c14e7487809b" (UID: "9cf76c9b-fa0c-4575-aeae-c14e7487809b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:54:55.630084 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.630015 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cf76c9b-fa0c-4575-aeae-c14e7487809b-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:55.632108 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.632079 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cf76c9b-fa0c-4575-aeae-c14e7487809b-kube-api-access-xzj68" pod "9cf76c9b-fa0c-4575-aeae-c14e7487809b" (UID: "9cf76c9b-fa0c-4575-aeae-c14e7487809b"). InnerVolumeSpecName "kube-api-access-xzj68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:54:55.632258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.632128 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-metrics-certs" pod "9cf76c9b-fa0c-4575-aeae-c14e7487809b" (UID: "9cf76c9b-fa0c-4575-aeae-c14e7487809b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:54:55.632258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.632104 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-maas-api-tls" pod "9cf76c9b-fa0c-4575-aeae-c14e7487809b" (UID: "9cf76c9b-fa0c-4575-aeae-c14e7487809b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:54:55.731524 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.731486 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:55.731524 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.731514 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cf76c9b-fa0c-4575-aeae-c14e7487809b-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:55.731524 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.731528 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzj68\" (UniqueName: \"kubernetes.io/projected/9cf76c9b-fa0c-4575-aeae-c14e7487809b-kube-api-access-xzj68\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:54:55.779151 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.779109 6999 scope.go:122] "RemoveContainer" containerID="981c4edb59f0defddc66ffd1764ce076a41a7d893f50a7123af0846f8fa28aaa" Oct 30 06:54:55.826611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.826572 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" event={"ID":"9cf76c9b-fa0c-4575-aeae-c14e7487809b","Type":"ContainerDied","Data":"981c4edb59f0defddc66ffd1764ce076a41a7d893f50a7123af0846f8fa28aaa"} Oct 30 06:54:55.826842 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.826714 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" event={"ID":"9cf76c9b-fa0c-4575-aeae-c14e7487809b","Type":"ContainerDied","Data":"ecec1ec0befe9da6ab142a11576dfa1c24b897da4a295b829a24c0af0fc074d6"} Oct 30 06:54:55.826842 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.826801 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw" Oct 30 06:54:55.878105 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.878073 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw"] Oct 30 06:54:55.884416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.884390 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-b66aad-74b67f98d6-hqdqw"] Oct 30 06:54:55.929525 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:55.929501 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:56.068713 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:56.068679 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:56.578624 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:56.578594 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:57.389006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:57.388970 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:57.531185 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:57.531145 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:57.609443 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:57.609414 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cf76c9b-fa0c-4575-aeae-c14e7487809b" path="/var/lib/kubelet/pods/9cf76c9b-fa0c-4575-aeae-c14e7487809b/volumes" Oct 30 06:54:57.690314 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:57.681609 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq"] Oct 30 06:54:57.690314 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:57.681867 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" podUID="f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" containerName="istio-proxy" containerID="cri-o://d1ccd3604e1835cb22be0eefc66f02dcc6e34371f5a24577b3acee0f011b186f" gracePeriod=30 Oct 30 06:54:57.750613 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:57.750574 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:57.892834 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:57.892798 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:54:59.638387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:54:59.638344 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:00.753538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:00.753504 6999 patch_prober.go:29] interesting pod/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:55:00.753983 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:00.753560 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" podUID="f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:55:01.435876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:01.435836 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:02.878449 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:02.878411 6999 generic.go:356] "Generic (PLEG): container finished" podID="f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" containerID="d1ccd3604e1835cb22be0eefc66f02dcc6e34371f5a24577b3acee0f011b186f" exitCode=0 Oct 30 06:55:02.878918 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:02.878544 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" event={"ID":"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb","Type":"ContainerDied","Data":"d1ccd3604e1835cb22be0eefc66f02dcc6e34371f5a24577b3acee0f011b186f"} Oct 30 06:55:02.972065 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:02.972038 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:55:03.024759 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.024728 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-data\") pod \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " Oct 30 06:55:03.025030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.024897 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-workload-certs\") pod \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " Oct 30 06:55:03.025030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.024939 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-token\" (UniqueName: \"kubernetes.io/projected/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-token\") pod \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " Oct 30 06:55:03.025030 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.024967 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-credential-socket\") pod \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " Oct 30 06:55:03.025250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.025032 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-envoy\") pod \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " Oct 30 06:55:03.025250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.025062 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-kube-api-access-vjxvz\" (UniqueName: \"kubernetes.io/projected/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-kube-api-access-vjxvz\") pod \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " Oct 30 06:55:03.025250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.025119 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-workload-socket\") pod \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " Oct 30 06:55:03.025250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.025174 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-data" pod "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" (UID: "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:03.025250 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.025226 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-podinfo\") pod \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " Oct 30 06:55:03.025504 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.025269 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istiod-ca-cert\") pod \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\" (UID: \"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb\") " Oct 30 06:55:03.025504 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.025391 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-workload-certs" pod "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" (UID: "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:03.026343 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.025782 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.026343 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.025813 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.026343 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.025782 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-workload-socket" pod "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" (UID: "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:03.026343 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.025818 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istiod-ca-cert" pod "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" (UID: "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:55:03.026343 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.026101 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-credential-socket" pod "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" (UID: "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:03.028162 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.028136 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-podinfo" pod "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" (UID: "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:55:03.028589 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.028567 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-kube-api-access-vjxvz" pod "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" (UID: "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb"). InnerVolumeSpecName "kube-api-access-vjxvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:55:03.030587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.030561 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-token" pod "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" (UID: "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:55:03.035054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.035032 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-envoy" pod "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" (UID: "f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:55:03.126736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.126700 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.126736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.126731 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.126736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.126742 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.126977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.126750 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.126977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.126759 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.126977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.126769 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjxvz\" (UniqueName: \"kubernetes.io/projected/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-kube-api-access-vjxvz\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.126977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.126777 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:55:03.280683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.280596 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:03.886907 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.886836 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" Oct 30 06:55:03.886907 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.886871 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq" event={"ID":"f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb","Type":"ContainerDied","Data":"4f91e353a53d91ef04e8d31d565c3667af442cba04c4036841189583de456ee9"} Oct 30 06:55:03.887408 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.886933 6999 scope.go:122] "RemoveContainer" containerID="d1ccd3604e1835cb22be0eefc66f02dcc6e34371f5a24577b3acee0f011b186f" Oct 30 06:55:03.954299 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.954270 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq"] Oct 30 06:55:03.966683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:03.966645 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-eaee88-openshift-default-5b98dc9889-8q4mq"] Oct 30 06:55:04.028841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:04.028805 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:04.150033 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:55:04.149879 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:55:04.175904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:04.175863 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:04.386587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:04.386555 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:05.130100 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:05.130055 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:05.610051 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:05.610027 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" path="/var/lib/kubelet/pods/f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb/volumes" Oct 30 06:55:06.162461 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:06.162433 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:07.020977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:07.020938 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:08.647486 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:08.647453 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:08.823491 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:08.823459 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:10.583384 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:10.583341 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:12.365568 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:12.365528 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:14.234336 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:14.234301 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:15.603308 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:15.603274 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:15.745826 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:15.745783 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:16.032166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:16.032129 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:16.838052 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:16.838025 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:16.979343 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:16.979316 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:17.738406 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:17.738375 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:19.446467 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:19.446435 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:21.227473 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:21.227442 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:22.407898 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:22.407866 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:22.549805 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:22.549768 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:22.988756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:22.988726 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:24.851762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:24.851728 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:26.718957 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:26.718916 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:28.516193 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:28.516160 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:30.286843 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:30.286807 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:32.118842 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:32.118811 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:33.912923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:33.912816 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:34.142005 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:55:34.141969 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:55:35.627628 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:35.627596 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:37.440506 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:37.440472 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:39.217686 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:39.217652 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:40.960840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:40.960806 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:42.420002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.419967 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj"] Oct 30 06:55:42.420565 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.420184 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" podUID="ead1e0b2-5bb1-451d-865d-2fc40e243987" containerName="maas-api" containerID="cri-o://c8732042a2c3397b07cb3b34abb1a7da87412211a2fe77f57813e9df7194a2ff" gracePeriod=30 Oct 30 06:55:42.519584 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.519538 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:42.746564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.746538 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:55:42.813953 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.813919 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ead1e0b2-5bb1-451d-865d-2fc40e243987-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead1e0b2-5bb1-451d-865d-2fc40e243987-maas-api-tls\") pod \"ead1e0b2-5bb1-451d-865d-2fc40e243987\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " Oct 30 06:55:42.814203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.814182 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead1e0b2-5bb1-451d-865d-2fc40e243987-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead1e0b2-5bb1-451d-865d-2fc40e243987-openshift-service-ca\") pod \"ead1e0b2-5bb1-451d-865d-2fc40e243987\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " Oct 30 06:55:42.814339 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.814321 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ead1e0b2-5bb1-451d-865d-2fc40e243987-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead1e0b2-5bb1-451d-865d-2fc40e243987-metrics-certs\") pod \"ead1e0b2-5bb1-451d-865d-2fc40e243987\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " Oct 30 06:55:42.814406 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.814362 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead1e0b2-5bb1-451d-865d-2fc40e243987-kube-api-access-rwz7k\" (UniqueName: \"kubernetes.io/projected/ead1e0b2-5bb1-451d-865d-2fc40e243987-kube-api-access-rwz7k\") pod \"ead1e0b2-5bb1-451d-865d-2fc40e243987\" (UID: \"ead1e0b2-5bb1-451d-865d-2fc40e243987\") " Oct 30 06:55:42.814720 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.814686 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead1e0b2-5bb1-451d-865d-2fc40e243987-openshift-service-ca" pod "ead1e0b2-5bb1-451d-865d-2fc40e243987" (UID: "ead1e0b2-5bb1-451d-865d-2fc40e243987"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:55:42.815943 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.815915 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead1e0b2-5bb1-451d-865d-2fc40e243987-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:55:42.818274 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.818207 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead1e0b2-5bb1-451d-865d-2fc40e243987-kube-api-access-rwz7k" pod "ead1e0b2-5bb1-451d-865d-2fc40e243987" (UID: "ead1e0b2-5bb1-451d-865d-2fc40e243987"). InnerVolumeSpecName "kube-api-access-rwz7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:55:42.818674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.818647 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead1e0b2-5bb1-451d-865d-2fc40e243987-maas-api-tls" pod "ead1e0b2-5bb1-451d-865d-2fc40e243987" (UID: "ead1e0b2-5bb1-451d-865d-2fc40e243987"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:55:42.819429 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.819405 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead1e0b2-5bb1-451d-865d-2fc40e243987-metrics-certs" pod "ead1e0b2-5bb1-451d-865d-2fc40e243987" (UID: "ead1e0b2-5bb1-451d-865d-2fc40e243987"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:55:42.917046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.917013 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ead1e0b2-5bb1-451d-865d-2fc40e243987-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:55:42.917046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.917041 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwz7k\" (UniqueName: \"kubernetes.io/projected/ead1e0b2-5bb1-451d-865d-2fc40e243987-kube-api-access-rwz7k\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:55:42.917046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:42.917050 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ead1e0b2-5bb1-451d-865d-2fc40e243987-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:55:43.046880 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:43.046784 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:43.158519 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:43.158487 6999 generic.go:356] "Generic (PLEG): container finished" podID="ead1e0b2-5bb1-451d-865d-2fc40e243987" containerID="c8732042a2c3397b07cb3b34abb1a7da87412211a2fe77f57813e9df7194a2ff" exitCode=0 Oct 30 06:55:43.158702 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:43.158529 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" Oct 30 06:55:43.158702 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:43.158529 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" event={"ID":"ead1e0b2-5bb1-451d-865d-2fc40e243987","Type":"ContainerDied","Data":"c8732042a2c3397b07cb3b34abb1a7da87412211a2fe77f57813e9df7194a2ff"} Oct 30 06:55:43.158702 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:43.158668 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj" event={"ID":"ead1e0b2-5bb1-451d-865d-2fc40e243987","Type":"ContainerDied","Data":"d21cc36fe1658ce0f9add4ab420a33724ffd2494ae1a7a03d776d818a5e46194"} Oct 30 06:55:43.158702 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:43.158701 6999 scope.go:122] "RemoveContainer" containerID="c8732042a2c3397b07cb3b34abb1a7da87412211a2fe77f57813e9df7194a2ff" Oct 30 06:55:43.170322 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:43.170266 6999 scope.go:122] "RemoveContainer" containerID="c8732042a2c3397b07cb3b34abb1a7da87412211a2fe77f57813e9df7194a2ff" Oct 30 06:55:43.170613 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:55:43.170589 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8732042a2c3397b07cb3b34abb1a7da87412211a2fe77f57813e9df7194a2ff\": container with ID starting with c8732042a2c3397b07cb3b34abb1a7da87412211a2fe77f57813e9df7194a2ff not found: ID does not exist" containerID="c8732042a2c3397b07cb3b34abb1a7da87412211a2fe77f57813e9df7194a2ff" Oct 30 06:55:43.170683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:43.170627 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8732042a2c3397b07cb3b34abb1a7da87412211a2fe77f57813e9df7194a2ff"} err="failed to get container status \"c8732042a2c3397b07cb3b34abb1a7da87412211a2fe77f57813e9df7194a2ff\": rpc error: code = NotFound desc = could not find container \"c8732042a2c3397b07cb3b34abb1a7da87412211a2fe77f57813e9df7194a2ff\": container with ID starting with c8732042a2c3397b07cb3b34abb1a7da87412211a2fe77f57813e9df7194a2ff not found: ID does not exist" Oct 30 06:55:43.191992 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:43.191974 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:43.212589 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:43.212564 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj"] Oct 30 06:55:43.219603 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:43.219583 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-88acce-54d9ddf887-hqltj"] Oct 30 06:55:43.331891 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:43.331821 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:43.609923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:43.609885 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead1e0b2-5bb1-451d-865d-2fc40e243987" path="/var/lib/kubelet/pods/ead1e0b2-5bb1-451d-865d-2fc40e243987/volumes" Oct 30 06:55:43.711697 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:43.711669 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:45.120775 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:45.120747 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:46.835924 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:46.835891 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:48.551359 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:48.551333 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:50.112416 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:50.112382 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:50.258997 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:50.258968 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:50.432200 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:50.432093 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:50.579009 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:50.578971 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:52.369863 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:52.369829 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:54.141881 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:54.141846 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:55.915229 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:55.915189 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:57.817301 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:57.817273 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:55:59.551778 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:55:59.551741 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:01.241749 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:01.241716 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:01.671723 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:01.671689 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:01.815333 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:01.815303 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:03.006877 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:03.006824 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:04.142945 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:56:04.142911 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:56:04.740075 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:04.740043 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:06.293956 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:06.293904 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:06.460705 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:06.460669 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:08.164450 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:08.164419 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:08.777852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:08.777818 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:09.938774 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:09.938737 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:11.744678 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:11.744605 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:13.519825 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:13.519794 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:15.316168 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:15.316132 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:17.167582 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:17.167507 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:18.994385 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:18.994345 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:20.711118 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:20.711086 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:22.478262 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:22.478195 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:24.243614 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:24.243583 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:26.042167 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:26.042133 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:27.825141 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:27.825110 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:29.577790 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:29.577756 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:31.275784 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:31.275753 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:32.044884 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:32.044849 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:33.096722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:33.096673 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:34.142937 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:56:34.142900 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:56:34.840174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:34.840144 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:36.621742 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:36.621710 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:38.302755 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:38.302720 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:38.436463 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:38.436429 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:40.138160 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:40.138127 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:41.576142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:41.576106 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52"] Oct 30 06:56:41.576641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:41.576414 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" podUID="db9631b2-8f37-4465-83e0-6c67404aeb5b" containerName="maas-api" containerID="cri-o://f72f9b32410e2fc5b5c385e4fe208f26f4f586f8c50c4336bdb36162f84e9fab" gracePeriod=30 Oct 30 06:56:41.668811 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:41.668779 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:41.902999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:41.902966 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:56:42.048196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.048163 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db9631b2-8f37-4465-83e0-6c67404aeb5b-kube-api-access-476bm\" (UniqueName: \"kubernetes.io/projected/db9631b2-8f37-4465-83e0-6c67404aeb5b-kube-api-access-476bm\") pod \"db9631b2-8f37-4465-83e0-6c67404aeb5b\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " Oct 30 06:56:42.048420 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.048347 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-maas-api-tls\") pod \"db9631b2-8f37-4465-83e0-6c67404aeb5b\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " Oct 30 06:56:42.048420 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.048384 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-metrics-certs\") pod \"db9631b2-8f37-4465-83e0-6c67404aeb5b\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " Oct 30 06:56:42.048539 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.048447 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db9631b2-8f37-4465-83e0-6c67404aeb5b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9631b2-8f37-4465-83e0-6c67404aeb5b-openshift-service-ca\") pod \"db9631b2-8f37-4465-83e0-6c67404aeb5b\" (UID: \"db9631b2-8f37-4465-83e0-6c67404aeb5b\") " Oct 30 06:56:42.049168 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.049141 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9631b2-8f37-4465-83e0-6c67404aeb5b-openshift-service-ca" pod "db9631b2-8f37-4465-83e0-6c67404aeb5b" (UID: "db9631b2-8f37-4465-83e0-6c67404aeb5b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:56:42.051016 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.050982 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9631b2-8f37-4465-83e0-6c67404aeb5b-kube-api-access-476bm" pod "db9631b2-8f37-4465-83e0-6c67404aeb5b" (UID: "db9631b2-8f37-4465-83e0-6c67404aeb5b"). InnerVolumeSpecName "kube-api-access-476bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:56:42.051379 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.051342 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-maas-api-tls" pod "db9631b2-8f37-4465-83e0-6c67404aeb5b" (UID: "db9631b2-8f37-4465-83e0-6c67404aeb5b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:56:42.051553 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.051529 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-metrics-certs" pod "db9631b2-8f37-4465-83e0-6c67404aeb5b" (UID: "db9631b2-8f37-4465-83e0-6c67404aeb5b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:56:42.149822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.149753 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db9631b2-8f37-4465-83e0-6c67404aeb5b-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:56:42.149822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.149778 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-476bm\" (UniqueName: \"kubernetes.io/projected/db9631b2-8f37-4465-83e0-6c67404aeb5b-kube-api-access-476bm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:56:42.149822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.149789 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:56:42.149822 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.149797 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db9631b2-8f37-4465-83e0-6c67404aeb5b-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:56:42.247760 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.247731 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:42.393447 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.393414 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:42.529841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.529770 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:42.601837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.601806 6999 generic.go:356] "Generic (PLEG): container finished" podID="db9631b2-8f37-4465-83e0-6c67404aeb5b" containerID="f72f9b32410e2fc5b5c385e4fe208f26f4f586f8c50c4336bdb36162f84e9fab" exitCode=0 Oct 30 06:56:42.602365 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.601853 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" Oct 30 06:56:42.602365 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.601871 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" event={"ID":"db9631b2-8f37-4465-83e0-6c67404aeb5b","Type":"ContainerDied","Data":"f72f9b32410e2fc5b5c385e4fe208f26f4f586f8c50c4336bdb36162f84e9fab"} Oct 30 06:56:42.602365 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.601917 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52" event={"ID":"db9631b2-8f37-4465-83e0-6c67404aeb5b","Type":"ContainerDied","Data":"c549e6ba061d565bbf89ed6f5dd2ab3381094b7da1cccb9be299857dee1e5d35"} Oct 30 06:56:42.602365 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.601944 6999 scope.go:122] "RemoveContainer" containerID="f72f9b32410e2fc5b5c385e4fe208f26f4f586f8c50c4336bdb36162f84e9fab" Oct 30 06:56:42.613551 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.613531 6999 scope.go:122] "RemoveContainer" containerID="f72f9b32410e2fc5b5c385e4fe208f26f4f586f8c50c4336bdb36162f84e9fab" Oct 30 06:56:42.613809 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:56:42.613789 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f72f9b32410e2fc5b5c385e4fe208f26f4f586f8c50c4336bdb36162f84e9fab\": container with ID starting with f72f9b32410e2fc5b5c385e4fe208f26f4f586f8c50c4336bdb36162f84e9fab not found: ID does not exist" containerID="f72f9b32410e2fc5b5c385e4fe208f26f4f586f8c50c4336bdb36162f84e9fab" Oct 30 06:56:42.613860 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.613817 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72f9b32410e2fc5b5c385e4fe208f26f4f586f8c50c4336bdb36162f84e9fab"} err="failed to get container status \"f72f9b32410e2fc5b5c385e4fe208f26f4f586f8c50c4336bdb36162f84e9fab\": rpc error: code = NotFound desc = could not find container \"f72f9b32410e2fc5b5c385e4fe208f26f4f586f8c50c4336bdb36162f84e9fab\": container with ID starting with f72f9b32410e2fc5b5c385e4fe208f26f4f586f8c50c4336bdb36162f84e9fab not found: ID does not exist" Oct 30 06:56:42.655401 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.655377 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52"] Oct 30 06:56:42.667396 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.667366 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c44730-6c98bccc4f-jrc52"] Oct 30 06:56:42.902349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:42.902316 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:43.615660 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:43.615627 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db9631b2-8f37-4465-83e0-6c67404aeb5b" path="/var/lib/kubelet/pods/db9631b2-8f37-4465-83e0-6c67404aeb5b/volumes" Oct 30 06:56:44.242897 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:44.242861 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:46.035914 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:46.035869 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:47.936804 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:47.936758 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:49.577538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:49.577492 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:49.712900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:49.712870 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:49.858208 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:49.858134 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:50.002641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:50.002607 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:50.830207 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:50.830170 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf"] Oct 30 06:56:50.901802 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:50.901733 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:51.652400 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:51.652360 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:53.452788 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:53.452750 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:55.187971 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:55.187937 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:55.352339 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:55.352307 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89"] Oct 30 06:56:55.426791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:55.426760 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:57.141411 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:57.141378 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:56:58.840364 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:56:58.840296 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:00.601474 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:00.601441 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:01.154679 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:01.154642 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:01.293468 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:01.293432 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:01.915904 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:01.915873 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:02.057715 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:02.057684 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:02.450656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:02.450624 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:04.145277 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:57:04.145166 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:57:04.230759 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:04.230718 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:05.523445 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:05.523417 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:05.860010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:05.859974 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" podUID="afadd843-8c76-4a9c-b193-90b1ff6560e7" containerName="main" containerID="cri-o://aaaa6efe9dd7c9bf7fee63662b72bf705ce8aa6d56a215538ebc3ab656613b8d" gracePeriod=45 Oct 30 06:57:05.918351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:05.918326 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:06.127772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.127747 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:06.160587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.160563 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:57:06.252455 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.252422 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-tmp-dir\") pod \"afadd843-8c76-4a9c-b193-90b1ff6560e7\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " Oct 30 06:57:06.252647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.252463 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-home\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-home\") pod \"afadd843-8c76-4a9c-b193-90b1ff6560e7\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " Oct 30 06:57:06.252647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.252497 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-model-cache\") pod \"afadd843-8c76-4a9c-b193-90b1ff6560e7\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " Oct 30 06:57:06.252647 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.252558 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afadd843-8c76-4a9c-b193-90b1ff6560e7-kube-api-access-lxmq7\" (UniqueName: \"kubernetes.io/projected/afadd843-8c76-4a9c-b193-90b1ff6560e7-kube-api-access-lxmq7\") pod \"afadd843-8c76-4a9c-b193-90b1ff6560e7\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " Oct 30 06:57:06.252838 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.252707 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afadd843-8c76-4a9c-b193-90b1ff6560e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/afadd843-8c76-4a9c-b193-90b1ff6560e7-tls-certs\") pod \"afadd843-8c76-4a9c-b193-90b1ff6560e7\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " Oct 30 06:57:06.252838 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.252755 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-tmp-dir" pod "afadd843-8c76-4a9c-b193-90b1ff6560e7" (UID: "afadd843-8c76-4a9c-b193-90b1ff6560e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:06.252838 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.252806 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-dshm\") pod \"afadd843-8c76-4a9c-b193-90b1ff6560e7\" (UID: \"afadd843-8c76-4a9c-b193-90b1ff6560e7\") " Oct 30 06:57:06.253006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.252834 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-home" pod "afadd843-8c76-4a9c-b193-90b1ff6560e7" (UID: "afadd843-8c76-4a9c-b193-90b1ff6560e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:06.253006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.252879 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-model-cache" pod "afadd843-8c76-4a9c-b193-90b1ff6560e7" (UID: "afadd843-8c76-4a9c-b193-90b1ff6560e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:06.253552 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.253529 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:06.253700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.253555 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:06.253700 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.253569 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:06.255105 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.255076 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-dshm" pod "afadd843-8c76-4a9c-b193-90b1ff6560e7" (UID: "afadd843-8c76-4a9c-b193-90b1ff6560e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:06.255281 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.255103 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afadd843-8c76-4a9c-b193-90b1ff6560e7-tls-certs" pod "afadd843-8c76-4a9c-b193-90b1ff6560e7" (UID: "afadd843-8c76-4a9c-b193-90b1ff6560e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:06.255281 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.255118 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afadd843-8c76-4a9c-b193-90b1ff6560e7-kube-api-access-lxmq7" pod "afadd843-8c76-4a9c-b193-90b1ff6560e7" (UID: "afadd843-8c76-4a9c-b193-90b1ff6560e7"). InnerVolumeSpecName "kube-api-access-lxmq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:06.354456 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.354419 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxmq7\" (UniqueName: \"kubernetes.io/projected/afadd843-8c76-4a9c-b193-90b1ff6560e7-kube-api-access-lxmq7\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:06.354456 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.354455 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afadd843-8c76-4a9c-b193-90b1ff6560e7-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:06.354660 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.354466 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afadd843-8c76-4a9c-b193-90b1ff6560e7-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:06.421413 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.421331 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:06.762757 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.762671 6999 generic.go:356] "Generic (PLEG): container finished" podID="afadd843-8c76-4a9c-b193-90b1ff6560e7" containerID="aaaa6efe9dd7c9bf7fee63662b72bf705ce8aa6d56a215538ebc3ab656613b8d" exitCode=0 Oct 30 06:57:06.762757 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.762740 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" Oct 30 06:57:06.763331 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.762772 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" event={"ID":"afadd843-8c76-4a9c-b193-90b1ff6560e7","Type":"ContainerDied","Data":"aaaa6efe9dd7c9bf7fee63662b72bf705ce8aa6d56a215538ebc3ab656613b8d"} Oct 30 06:57:06.763331 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.762827 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf" event={"ID":"afadd843-8c76-4a9c-b193-90b1ff6560e7","Type":"ContainerDied","Data":"51497b3d46e7fde5d1ecfafc0b1975dee739f8d3059178ff445c2b6388d55893"} Oct 30 06:57:06.763331 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.762851 6999 scope.go:122] "RemoveContainer" containerID="aaaa6efe9dd7c9bf7fee63662b72bf705ce8aa6d56a215538ebc3ab656613b8d" Oct 30 06:57:06.774088 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.774071 6999 scope.go:122] "RemoveContainer" containerID="aaaa6efe9dd7c9bf7fee63662b72bf705ce8aa6d56a215538ebc3ab656613b8d" Oct 30 06:57:06.774395 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:57:06.774373 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaaa6efe9dd7c9bf7fee63662b72bf705ce8aa6d56a215538ebc3ab656613b8d\": container with ID starting with aaaa6efe9dd7c9bf7fee63662b72bf705ce8aa6d56a215538ebc3ab656613b8d not found: ID does not exist" containerID="aaaa6efe9dd7c9bf7fee63662b72bf705ce8aa6d56a215538ebc3ab656613b8d" Oct 30 06:57:06.774495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.774404 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaaa6efe9dd7c9bf7fee63662b72bf705ce8aa6d56a215538ebc3ab656613b8d"} err="failed to get container status \"aaaa6efe9dd7c9bf7fee63662b72bf705ce8aa6d56a215538ebc3ab656613b8d\": rpc error: code = NotFound desc = could not find container \"aaaa6efe9dd7c9bf7fee63662b72bf705ce8aa6d56a215538ebc3ab656613b8d\": container with ID starting with aaaa6efe9dd7c9bf7fee63662b72bf705ce8aa6d56a215538ebc3ab656613b8d not found: ID does not exist" Oct 30 06:57:06.814623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.814601 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf"] Oct 30 06:57:06.827103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:06.827083 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a5f459/test-model-a5f459-kserve-68d687fc8c-g8gmf"] Oct 30 06:57:07.610061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:07.610027 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afadd843-8c76-4a9c-b193-90b1ff6560e7" path="/var/lib/kubelet/pods/afadd843-8c76-4a9c-b193-90b1ff6560e7/volumes" Oct 30 06:57:07.881910 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:07.881806 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:08.896550 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:08.896518 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:09.733028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:09.732984 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:10.374412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.374378 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" podUID="1c9f6978-1b17-4e4d-a28b-aed920888f5f" containerName="main" containerID="cri-o://0653ca529f2eb563ba821addf554ccb4bc4695571af96e8461afce539e541d82" gracePeriod=45 Oct 30 06:57:10.648043 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.648016 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:57:10.789767 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.789732 6999 generic.go:356] "Generic (PLEG): container finished" podID="1c9f6978-1b17-4e4d-a28b-aed920888f5f" containerID="0653ca529f2eb563ba821addf554ccb4bc4695571af96e8461afce539e541d82" exitCode=0 Oct 30 06:57:10.789948 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.789778 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" Oct 30 06:57:10.789948 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.789813 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" event={"ID":"1c9f6978-1b17-4e4d-a28b-aed920888f5f","Type":"ContainerDied","Data":"0653ca529f2eb563ba821addf554ccb4bc4695571af96e8461afce539e541d82"} Oct 30 06:57:10.789948 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.789853 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89" event={"ID":"1c9f6978-1b17-4e4d-a28b-aed920888f5f","Type":"ContainerDied","Data":"b763b363d3ed993a323182d147d7b0547808891732f8b9ae8b15eadcc32918bb"} Oct 30 06:57:10.789948 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.789874 6999 scope.go:122] "RemoveContainer" containerID="0653ca529f2eb563ba821addf554ccb4bc4695571af96e8461afce539e541d82" Oct 30 06:57:10.802545 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.802521 6999 scope.go:122] "RemoveContainer" containerID="0653ca529f2eb563ba821addf554ccb4bc4695571af96e8461afce539e541d82" Oct 30 06:57:10.802918 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:57:10.802895 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0653ca529f2eb563ba821addf554ccb4bc4695571af96e8461afce539e541d82\": container with ID starting with 0653ca529f2eb563ba821addf554ccb4bc4695571af96e8461afce539e541d82 not found: ID does not exist" containerID="0653ca529f2eb563ba821addf554ccb4bc4695571af96e8461afce539e541d82" Oct 30 06:57:10.803018 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.802926 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0653ca529f2eb563ba821addf554ccb4bc4695571af96e8461afce539e541d82"} err="failed to get container status \"0653ca529f2eb563ba821addf554ccb4bc4695571af96e8461afce539e541d82\": rpc error: code = NotFound desc = could not find container \"0653ca529f2eb563ba821addf554ccb4bc4695571af96e8461afce539e541d82\": container with ID starting with 0653ca529f2eb563ba821addf554ccb4bc4695571af96e8461afce539e541d82 not found: ID does not exist" Oct 30 06:57:10.804278 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.804257 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9f6978-1b17-4e4d-a28b-aed920888f5f-kube-api-access-jl28h\" (UniqueName: \"kubernetes.io/projected/1c9f6978-1b17-4e4d-a28b-aed920888f5f-kube-api-access-jl28h\") pod \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " Oct 30 06:57:10.804393 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.804377 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-dshm\") pod \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " Oct 30 06:57:10.804459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.804435 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c9f6978-1b17-4e4d-a28b-aed920888f5f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9f6978-1b17-4e4d-a28b-aed920888f5f-tls-certs\") pod \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " Oct 30 06:57:10.804563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.804548 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-tmp-dir\") pod \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " Oct 30 06:57:10.804644 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.804631 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-model-cache\") pod \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " Oct 30 06:57:10.804708 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.804664 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-home\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-home\") pod \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\" (UID: \"1c9f6978-1b17-4e4d-a28b-aed920888f5f\") " Oct 30 06:57:10.804905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.804829 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-tmp-dir" pod "1c9f6978-1b17-4e4d-a28b-aed920888f5f" (UID: "1c9f6978-1b17-4e4d-a28b-aed920888f5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:10.805226 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.804934 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-model-cache" pod "1c9f6978-1b17-4e4d-a28b-aed920888f5f" (UID: "1c9f6978-1b17-4e4d-a28b-aed920888f5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:10.805226 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.805020 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-home" pod "1c9f6978-1b17-4e4d-a28b-aed920888f5f" (UID: "1c9f6978-1b17-4e4d-a28b-aed920888f5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:10.806755 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.806732 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-dshm" pod "1c9f6978-1b17-4e4d-a28b-aed920888f5f" (UID: "1c9f6978-1b17-4e4d-a28b-aed920888f5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:10.806848 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.806778 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c9f6978-1b17-4e4d-a28b-aed920888f5f-tls-certs" pod "1c9f6978-1b17-4e4d-a28b-aed920888f5f" (UID: "1c9f6978-1b17-4e4d-a28b-aed920888f5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:10.806848 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.806795 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9f6978-1b17-4e4d-a28b-aed920888f5f-kube-api-access-jl28h" pod "1c9f6978-1b17-4e4d-a28b-aed920888f5f" (UID: "1c9f6978-1b17-4e4d-a28b-aed920888f5f"). InnerVolumeSpecName "kube-api-access-jl28h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:10.906074 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.905994 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:10.906074 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.906024 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c9f6978-1b17-4e4d-a28b-aed920888f5f-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:10.906074 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.906034 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:10.906074 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.906042 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:10.906074 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.906049 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c9f6978-1b17-4e4d-a28b-aed920888f5f-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:10.906074 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:10.906058 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl28h\" (UniqueName: \"kubernetes.io/projected/1c9f6978-1b17-4e4d-a28b-aed920888f5f-kube-api-access-jl28h\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:11.109664 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.109635 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z"] Oct 30 06:57:11.109837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.109821 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" podUID="3162d9da-8ad6-4272-8374-af59367f2884" containerName="maas-api" containerID="cri-o://ea71ee2213e1a04bc27d9726704b370337665662fd7e7966c013b37dc6d496b7" gracePeriod=30 Oct 30 06:57:11.151639 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.151603 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89"] Oct 30 06:57:11.155619 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.155585 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-62efb8/test-model-62efb8-kserve-665f8c8757-5fx89"] Oct 30 06:57:11.200708 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.200620 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:11.432188 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.432159 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:57:11.513625 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.513544 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3162d9da-8ad6-4272-8374-af59367f2884-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3162d9da-8ad6-4272-8374-af59367f2884-openshift-service-ca\") pod \"3162d9da-8ad6-4272-8374-af59367f2884\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " Oct 30 06:57:11.513798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.513685 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-maas-api-tls\") pod \"3162d9da-8ad6-4272-8374-af59367f2884\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " Oct 30 06:57:11.513798 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.513725 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-metrics-certs\") pod \"3162d9da-8ad6-4272-8374-af59367f2884\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " Oct 30 06:57:11.513931 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.513805 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3162d9da-8ad6-4272-8374-af59367f2884-kube-api-access-h5k4t\" (UniqueName: \"kubernetes.io/projected/3162d9da-8ad6-4272-8374-af59367f2884-kube-api-access-h5k4t\") pod \"3162d9da-8ad6-4272-8374-af59367f2884\" (UID: \"3162d9da-8ad6-4272-8374-af59367f2884\") " Oct 30 06:57:11.513994 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.513930 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3162d9da-8ad6-4272-8374-af59367f2884-openshift-service-ca" pod "3162d9da-8ad6-4272-8374-af59367f2884" (UID: "3162d9da-8ad6-4272-8374-af59367f2884"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:11.514731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.514703 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3162d9da-8ad6-4272-8374-af59367f2884-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:11.516415 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.516374 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-metrics-certs" pod "3162d9da-8ad6-4272-8374-af59367f2884" (UID: "3162d9da-8ad6-4272-8374-af59367f2884"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:11.516523 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.516449 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-maas-api-tls" pod "3162d9da-8ad6-4272-8374-af59367f2884" (UID: "3162d9da-8ad6-4272-8374-af59367f2884"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:11.517085 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.517065 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3162d9da-8ad6-4272-8374-af59367f2884-kube-api-access-h5k4t" pod "3162d9da-8ad6-4272-8374-af59367f2884" (UID: "3162d9da-8ad6-4272-8374-af59367f2884"). InnerVolumeSpecName "kube-api-access-h5k4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:11.615418 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.615378 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c9f6978-1b17-4e4d-a28b-aed920888f5f" path="/var/lib/kubelet/pods/1c9f6978-1b17-4e4d-a28b-aed920888f5f/volumes" Oct 30 06:57:11.617035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.616552 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:11.617035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.616583 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3162d9da-8ad6-4272-8374-af59367f2884-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:11.617035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.616598 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5k4t\" (UniqueName: \"kubernetes.io/projected/3162d9da-8ad6-4272-8374-af59367f2884-kube-api-access-h5k4t\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:11.797670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.797585 6999 generic.go:356] "Generic (PLEG): container finished" podID="3162d9da-8ad6-4272-8374-af59367f2884" containerID="ea71ee2213e1a04bc27d9726704b370337665662fd7e7966c013b37dc6d496b7" exitCode=0 Oct 30 06:57:11.797670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.797639 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" Oct 30 06:57:11.797927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.797673 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" event={"ID":"3162d9da-8ad6-4272-8374-af59367f2884","Type":"ContainerDied","Data":"ea71ee2213e1a04bc27d9726704b370337665662fd7e7966c013b37dc6d496b7"} Oct 30 06:57:11.797927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.797726 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z" event={"ID":"3162d9da-8ad6-4272-8374-af59367f2884","Type":"ContainerDied","Data":"d73dd240edc3e6e65a414fb60e53652f8d5c51ac8c033eda64a249b8bea6fada"} Oct 30 06:57:11.797927 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.797752 6999 scope.go:122] "RemoveContainer" containerID="ea71ee2213e1a04bc27d9726704b370337665662fd7e7966c013b37dc6d496b7" Oct 30 06:57:11.807696 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.807680 6999 scope.go:122] "RemoveContainer" containerID="ea71ee2213e1a04bc27d9726704b370337665662fd7e7966c013b37dc6d496b7" Oct 30 06:57:11.807974 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:57:11.807946 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea71ee2213e1a04bc27d9726704b370337665662fd7e7966c013b37dc6d496b7\": container with ID starting with ea71ee2213e1a04bc27d9726704b370337665662fd7e7966c013b37dc6d496b7 not found: ID does not exist" containerID="ea71ee2213e1a04bc27d9726704b370337665662fd7e7966c013b37dc6d496b7" Oct 30 06:57:11.808027 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.807972 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea71ee2213e1a04bc27d9726704b370337665662fd7e7966c013b37dc6d496b7"} err="failed to get container status \"ea71ee2213e1a04bc27d9726704b370337665662fd7e7966c013b37dc6d496b7\": rpc error: code = NotFound desc = could not find container \"ea71ee2213e1a04bc27d9726704b370337665662fd7e7966c013b37dc6d496b7\": container with ID starting with ea71ee2213e1a04bc27d9726704b370337665662fd7e7966c013b37dc6d496b7 not found: ID does not exist" Oct 30 06:57:11.814788 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.814769 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:11.852443 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.852418 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z"] Oct 30 06:57:11.864013 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.863994 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-4f3973-6f7cf994cc-tjm8z"] Oct 30 06:57:11.959055 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:11.959000 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:12.453919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:12.453884 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:13.552182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:13.552145 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:13.610645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:13.610618 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3162d9da-8ad6-4272-8374-af59367f2884" path="/var/lib/kubelet/pods/3162d9da-8ad6-4272-8374-af59367f2884/volumes" Oct 30 06:57:14.093800 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:14.093757 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:14.317266 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:14.317191 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:14.737178 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:14.737148 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:16.140651 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:16.140621 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:17.933674 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:17.933640 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:19.431266 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:19.424190 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6"] Oct 30 06:57:19.494836 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:19.494804 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:19.775627 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:19.775542 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:19.922746 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:19.922707 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:20.188609 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:20.188573 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:21.836016 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:21.835973 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:21.979986 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:21.979945 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:23.082226 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:23.082192 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:23.220586 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:23.220559 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:23.599807 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:23.599773 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:23.980000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:23.979967 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:25.916115 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:25.916088 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:27.736562 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:27.736527 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:29.430452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:29.430418 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:30.314658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:30.314627 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:30.456612 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:30.456575 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:31.065182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:31.065147 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:31.206387 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:31.206352 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:32.531902 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.531860 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj"] Oct 30 06:57:32.532307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.532067 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" podUID="4f4c6e9e-70db-47c3-82f5-e1798c4e6009" containerName="maas-api" containerID="cri-o://1234af9cdf44f98256e88e99f7802582d988f0524cc6c8508082b4848c5ac0bc" gracePeriod=30 Oct 30 06:57:32.642037 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.641999 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:32.781721 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.781642 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:32.834891 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.834863 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:57:32.897969 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.897940 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-maas-api-tls\") pod \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " Oct 30 06:57:32.898170 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.897987 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-kube-api-access-tr9dr\" (UniqueName: \"kubernetes.io/projected/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-kube-api-access-tr9dr\") pod \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " Oct 30 06:57:32.898170 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.898016 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-openshift-service-ca\") pod \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " Oct 30 06:57:32.898170 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.898118 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-metrics-certs\") pod \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\" (UID: \"4f4c6e9e-70db-47c3-82f5-e1798c4e6009\") " Oct 30 06:57:32.898476 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.898447 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-openshift-service-ca" pod "4f4c6e9e-70db-47c3-82f5-e1798c4e6009" (UID: "4f4c6e9e-70db-47c3-82f5-e1798c4e6009"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:32.898926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.898764 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:32.900503 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.900477 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-kube-api-access-tr9dr" pod "4f4c6e9e-70db-47c3-82f5-e1798c4e6009" (UID: "4f4c6e9e-70db-47c3-82f5-e1798c4e6009"). InnerVolumeSpecName "kube-api-access-tr9dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:32.900750 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.900735 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-metrics-certs" pod "4f4c6e9e-70db-47c3-82f5-e1798c4e6009" (UID: "4f4c6e9e-70db-47c3-82f5-e1798c4e6009"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:32.900837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.900811 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-maas-api-tls" pod "4f4c6e9e-70db-47c3-82f5-e1798c4e6009" (UID: "4f4c6e9e-70db-47c3-82f5-e1798c4e6009"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:32.934610 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.934578 6999 generic.go:356] "Generic (PLEG): container finished" podID="4f4c6e9e-70db-47c3-82f5-e1798c4e6009" containerID="1234af9cdf44f98256e88e99f7802582d988f0524cc6c8508082b4848c5ac0bc" exitCode=0 Oct 30 06:57:32.934729 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.934624 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" Oct 30 06:57:32.934729 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.934657 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" event={"ID":"4f4c6e9e-70db-47c3-82f5-e1798c4e6009","Type":"ContainerDied","Data":"1234af9cdf44f98256e88e99f7802582d988f0524cc6c8508082b4848c5ac0bc"} Oct 30 06:57:32.934729 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.934714 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj" event={"ID":"4f4c6e9e-70db-47c3-82f5-e1798c4e6009","Type":"ContainerDied","Data":"aecdd7850cedbf7886d3ee08c4b6fd74d1f97c0b66842a5673331a26e7b70779"} Oct 30 06:57:32.934860 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.934736 6999 scope.go:122] "RemoveContainer" containerID="1234af9cdf44f98256e88e99f7802582d988f0524cc6c8508082b4848c5ac0bc" Oct 30 06:57:32.948722 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.948702 6999 scope.go:122] "RemoveContainer" containerID="1234af9cdf44f98256e88e99f7802582d988f0524cc6c8508082b4848c5ac0bc" Oct 30 06:57:32.948999 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:57:32.948971 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1234af9cdf44f98256e88e99f7802582d988f0524cc6c8508082b4848c5ac0bc\": container with ID starting with 1234af9cdf44f98256e88e99f7802582d988f0524cc6c8508082b4848c5ac0bc not found: ID does not exist" containerID="1234af9cdf44f98256e88e99f7802582d988f0524cc6c8508082b4848c5ac0bc" Oct 30 06:57:32.949091 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.949002 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1234af9cdf44f98256e88e99f7802582d988f0524cc6c8508082b4848c5ac0bc"} err="failed to get container status \"1234af9cdf44f98256e88e99f7802582d988f0524cc6c8508082b4848c5ac0bc\": rpc error: code = NotFound desc = could not find container \"1234af9cdf44f98256e88e99f7802582d988f0524cc6c8508082b4848c5ac0bc\": container with ID starting with 1234af9cdf44f98256e88e99f7802582d988f0524cc6c8508082b4848c5ac0bc not found: ID does not exist" Oct 30 06:57:32.986961 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.986934 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj"] Oct 30 06:57:32.991797 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:32.991776 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-b9ff93-65f967b5cc-lvsbj"] Oct 30 06:57:33.000076 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:33.000055 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr9dr\" (UniqueName: \"kubernetes.io/projected/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-kube-api-access-tr9dr\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:33.000177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:33.000078 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:33.000177 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:33.000088 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f4c6e9e-70db-47c3-82f5-e1798c4e6009-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:33.212427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:33.212367 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:33.359617 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:33.359588 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:33.619756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:33.619725 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f4c6e9e-70db-47c3-82f5-e1798c4e6009" path="/var/lib/kubelet/pods/4f4c6e9e-70db-47c3-82f5-e1798c4e6009/volumes" Oct 30 06:57:33.953624 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:33.953550 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:34.147044 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:57:34.147000 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:57:34.456565 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.456525 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" podUID="f79fd615-06d2-4a53-85cc-0c7da6834671" containerName="main" containerID="cri-o://c1454e9c62febbcae6f79a3fed2d6288b58561325bafe7b6ea4072814df1d25b" gracePeriod=45 Oct 30 06:57:34.479557 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.479531 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:34.618860 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.618827 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:34.743253 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.743225 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:57:34.826252 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.826204 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-model-cache\") pod \"f79fd615-06d2-4a53-85cc-0c7da6834671\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " Oct 30 06:57:34.826414 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.826312 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f79fd615-06d2-4a53-85cc-0c7da6834671-tls-certs\" (UniqueName: \"kubernetes.io/secret/f79fd615-06d2-4a53-85cc-0c7da6834671-tls-certs\") pod \"f79fd615-06d2-4a53-85cc-0c7da6834671\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " Oct 30 06:57:34.826414 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.826368 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-tmp-dir\") pod \"f79fd615-06d2-4a53-85cc-0c7da6834671\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " Oct 30 06:57:34.826414 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.826393 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79fd615-06d2-4a53-85cc-0c7da6834671-kube-api-access-pdbwb\" (UniqueName: \"kubernetes.io/projected/f79fd615-06d2-4a53-85cc-0c7da6834671-kube-api-access-pdbwb\") pod \"f79fd615-06d2-4a53-85cc-0c7da6834671\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " Oct 30 06:57:34.826596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.826465 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-model-cache" pod "f79fd615-06d2-4a53-85cc-0c7da6834671" (UID: "f79fd615-06d2-4a53-85cc-0c7da6834671"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:34.826596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.826520 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-home\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-home\") pod \"f79fd615-06d2-4a53-85cc-0c7da6834671\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " Oct 30 06:57:34.826693 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.826608 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-tmp-dir" pod "f79fd615-06d2-4a53-85cc-0c7da6834671" (UID: "f79fd615-06d2-4a53-85cc-0c7da6834671"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:34.826693 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.826633 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-dshm\") pod \"f79fd615-06d2-4a53-85cc-0c7da6834671\" (UID: \"f79fd615-06d2-4a53-85cc-0c7da6834671\") " Oct 30 06:57:34.826834 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.826811 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-home" pod "f79fd615-06d2-4a53-85cc-0c7da6834671" (UID: "f79fd615-06d2-4a53-85cc-0c7da6834671"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:34.827119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.827100 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:34.827119 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.827121 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:34.827334 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.827133 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:34.828786 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.828761 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f79fd615-06d2-4a53-85cc-0c7da6834671-tls-certs" pod "f79fd615-06d2-4a53-85cc-0c7da6834671" (UID: "f79fd615-06d2-4a53-85cc-0c7da6834671"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:34.828895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.828802 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-dshm" pod "f79fd615-06d2-4a53-85cc-0c7da6834671" (UID: "f79fd615-06d2-4a53-85cc-0c7da6834671"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:57:34.828895 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.828829 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79fd615-06d2-4a53-85cc-0c7da6834671-kube-api-access-pdbwb" pod "f79fd615-06d2-4a53-85cc-0c7da6834671" (UID: "f79fd615-06d2-4a53-85cc-0c7da6834671"). InnerVolumeSpecName "kube-api-access-pdbwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:34.931238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.929189 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f79fd615-06d2-4a53-85cc-0c7da6834671-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:34.931238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.929245 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f79fd615-06d2-4a53-85cc-0c7da6834671-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:34.931238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.929264 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdbwb\" (UniqueName: \"kubernetes.io/projected/f79fd615-06d2-4a53-85cc-0c7da6834671-kube-api-access-pdbwb\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:34.954939 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.954897 6999 generic.go:356] "Generic (PLEG): container finished" podID="f79fd615-06d2-4a53-85cc-0c7da6834671" containerID="c1454e9c62febbcae6f79a3fed2d6288b58561325bafe7b6ea4072814df1d25b" exitCode=0 Oct 30 06:57:34.955128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.955010 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" event={"ID":"f79fd615-06d2-4a53-85cc-0c7da6834671","Type":"ContainerDied","Data":"c1454e9c62febbcae6f79a3fed2d6288b58561325bafe7b6ea4072814df1d25b"} Oct 30 06:57:34.955128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.955046 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" event={"ID":"f79fd615-06d2-4a53-85cc-0c7da6834671","Type":"ContainerDied","Data":"78ab806514ad9aede666079a0dbfcd3ee7184f6da84d3f421aa176dfb3b38a01"} Oct 30 06:57:34.955128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.955067 6999 scope.go:122] "RemoveContainer" containerID="c1454e9c62febbcae6f79a3fed2d6288b58561325bafe7b6ea4072814df1d25b" Oct 30 06:57:34.955396 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.955245 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6" Oct 30 06:57:34.978689 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.978666 6999 scope.go:122] "RemoveContainer" containerID="c1454e9c62febbcae6f79a3fed2d6288b58561325bafe7b6ea4072814df1d25b" Oct 30 06:57:34.997351 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:57:34.997311 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1454e9c62febbcae6f79a3fed2d6288b58561325bafe7b6ea4072814df1d25b\": container with ID starting with c1454e9c62febbcae6f79a3fed2d6288b58561325bafe7b6ea4072814df1d25b not found: ID does not exist" containerID="c1454e9c62febbcae6f79a3fed2d6288b58561325bafe7b6ea4072814df1d25b" Oct 30 06:57:34.997505 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:34.997354 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1454e9c62febbcae6f79a3fed2d6288b58561325bafe7b6ea4072814df1d25b"} err="failed to get container status \"c1454e9c62febbcae6f79a3fed2d6288b58561325bafe7b6ea4072814df1d25b\": rpc error: code = NotFound desc = could not find container \"c1454e9c62febbcae6f79a3fed2d6288b58561325bafe7b6ea4072814df1d25b\": container with ID starting with c1454e9c62febbcae6f79a3fed2d6288b58561325bafe7b6ea4072814df1d25b not found: ID does not exist" Oct 30 06:57:35.015641 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:35.015610 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6"] Oct 30 06:57:35.020189 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:35.020165 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-89f101/test-model-89f101-kserve-654f586f65-4rjn6"] Oct 30 06:57:35.322763 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:35.322680 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:35.608452 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:35.608430 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f79fd615-06d2-4a53-85cc-0c7da6834671" path="/var/lib/kubelet/pods/f79fd615-06d2-4a53-85cc-0c7da6834671/volumes" Oct 30 06:57:36.691866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:36.691834 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5"] Oct 30 06:57:36.692651 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:36.692017 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" podUID="7dce711f-1134-46f0-ace9-ff3080839ede" containerName="maas-api" containerID="cri-o://12aa9e97abe4abc42ce31da04f5564a4b221b62b2ac3dd352d92dfb07d2c6d7d" gracePeriod=30 Oct 30 06:57:36.789052 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:36.789020 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:37.183944 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.183919 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:57:37.254069 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.253999 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dce711f-1134-46f0-ace9-ff3080839ede-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dce711f-1134-46f0-ace9-ff3080839ede-maas-api-tls\") pod \"7dce711f-1134-46f0-ace9-ff3080839ede\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " Oct 30 06:57:37.254255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.254112 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dce711f-1134-46f0-ace9-ff3080839ede-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dce711f-1134-46f0-ace9-ff3080839ede-openshift-service-ca\") pod \"7dce711f-1134-46f0-ace9-ff3080839ede\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " Oct 30 06:57:37.254255 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.254209 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7dce711f-1134-46f0-ace9-ff3080839ede-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dce711f-1134-46f0-ace9-ff3080839ede-metrics-certs\") pod \"7dce711f-1134-46f0-ace9-ff3080839ede\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " Oct 30 06:57:37.254396 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.254381 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dce711f-1134-46f0-ace9-ff3080839ede-kube-api-access-pb5hs\" (UniqueName: \"kubernetes.io/projected/7dce711f-1134-46f0-ace9-ff3080839ede-kube-api-access-pb5hs\") pod \"7dce711f-1134-46f0-ace9-ff3080839ede\" (UID: \"7dce711f-1134-46f0-ace9-ff3080839ede\") " Oct 30 06:57:37.254578 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.254551 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dce711f-1134-46f0-ace9-ff3080839ede-openshift-service-ca" pod "7dce711f-1134-46f0-ace9-ff3080839ede" (UID: "7dce711f-1134-46f0-ace9-ff3080839ede"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:37.255135 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.255082 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dce711f-1134-46f0-ace9-ff3080839ede-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.256485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.256459 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dce711f-1134-46f0-ace9-ff3080839ede-metrics-certs" pod "7dce711f-1134-46f0-ace9-ff3080839ede" (UID: "7dce711f-1134-46f0-ace9-ff3080839ede"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:37.256485 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.256474 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dce711f-1134-46f0-ace9-ff3080839ede-maas-api-tls" pod "7dce711f-1134-46f0-ace9-ff3080839ede" (UID: "7dce711f-1134-46f0-ace9-ff3080839ede"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:37.256630 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.256616 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dce711f-1134-46f0-ace9-ff3080839ede-kube-api-access-pb5hs" pod "7dce711f-1134-46f0-ace9-ff3080839ede" (UID: "7dce711f-1134-46f0-ace9-ff3080839ede"). InnerVolumeSpecName "kube-api-access-pb5hs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:37.323054 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.323020 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:37.356637 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.356607 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dce711f-1134-46f0-ace9-ff3080839ede-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.356637 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.356632 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7dce711f-1134-46f0-ace9-ff3080839ede-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.356782 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.356646 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb5hs\" (UniqueName: \"kubernetes.io/projected/7dce711f-1134-46f0-ace9-ff3080839ede-kube-api-access-pb5hs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:37.473788 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.473736 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:37.957510 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.957433 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:37.977938 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.977910 6999 generic.go:356] "Generic (PLEG): container finished" podID="7dce711f-1134-46f0-ace9-ff3080839ede" containerID="12aa9e97abe4abc42ce31da04f5564a4b221b62b2ac3dd352d92dfb07d2c6d7d" exitCode=0 Oct 30 06:57:37.978052 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.977958 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" Oct 30 06:57:37.978103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.977959 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" event={"ID":"7dce711f-1134-46f0-ace9-ff3080839ede","Type":"ContainerDied","Data":"12aa9e97abe4abc42ce31da04f5564a4b221b62b2ac3dd352d92dfb07d2c6d7d"} Oct 30 06:57:37.978103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.978096 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5" event={"ID":"7dce711f-1134-46f0-ace9-ff3080839ede","Type":"ContainerDied","Data":"55e49249a37ffeb49df648217259cf88ca90867ff50b0ea25707482bc3a8b1de"} Oct 30 06:57:37.978166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.978126 6999 scope.go:122] "RemoveContainer" containerID="12aa9e97abe4abc42ce31da04f5564a4b221b62b2ac3dd352d92dfb07d2c6d7d" Oct 30 06:57:37.987537 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.987518 6999 scope.go:122] "RemoveContainer" containerID="12aa9e97abe4abc42ce31da04f5564a4b221b62b2ac3dd352d92dfb07d2c6d7d" Oct 30 06:57:37.987850 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:57:37.987825 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12aa9e97abe4abc42ce31da04f5564a4b221b62b2ac3dd352d92dfb07d2c6d7d\": container with ID starting with 12aa9e97abe4abc42ce31da04f5564a4b221b62b2ac3dd352d92dfb07d2c6d7d not found: ID does not exist" containerID="12aa9e97abe4abc42ce31da04f5564a4b221b62b2ac3dd352d92dfb07d2c6d7d" Oct 30 06:57:37.987902 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:37.987859 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12aa9e97abe4abc42ce31da04f5564a4b221b62b2ac3dd352d92dfb07d2c6d7d"} err="failed to get container status \"12aa9e97abe4abc42ce31da04f5564a4b221b62b2ac3dd352d92dfb07d2c6d7d\": rpc error: code = NotFound desc = could not find container \"12aa9e97abe4abc42ce31da04f5564a4b221b62b2ac3dd352d92dfb07d2c6d7d\": container with ID starting with 12aa9e97abe4abc42ce31da04f5564a4b221b62b2ac3dd352d92dfb07d2c6d7d not found: ID does not exist" Oct 30 06:57:38.031597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:38.031573 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5"] Oct 30 06:57:38.044027 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:38.044004 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-62efb8-6ddb88c8fb-ppqf5"] Oct 30 06:57:39.433010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:39.432974 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:39.609693 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:39.609657 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dce711f-1134-46f0-ace9-ff3080839ede" path="/var/lib/kubelet/pods/7dce711f-1134-46f0-ace9-ff3080839ede/volumes" Oct 30 06:57:40.665150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:40.665122 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:40.819378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:40.819345 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:40.991000 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:40.990916 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:41.230366 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:41.230336 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:41.491265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:41.491237 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:43.122727 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:43.122697 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:44.614022 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:44.613988 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:44.752287 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:44.752203 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:44.910564 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:44.910478 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:45.054920 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:45.054887 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:46.448355 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:46.448321 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm"] Oct 30 06:57:46.567621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:46.567584 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:46.829640 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:46.829561 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:48.828087 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:48.828058 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:50.694359 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:50.694314 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:52.329776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:52.329746 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:52.473525 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:52.473493 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:52.616047 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:52.616024 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:54.217965 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:54.217930 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:55.666460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:55.666428 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp"] Oct 30 06:57:55.666909 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:55.666675 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" podUID="e7b70aac-3036-44cb-ae26-d4e0bea0a45e" containerName="maas-api" containerID="cri-o://59b374922932d63b6fc83bf852d5f391d5512039c32af75ac387cf330cf471e6" gracePeriod=30 Oct 30 06:57:55.771867 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:55.771824 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:55.917629 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:55.917600 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:56.021197 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.021171 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:57:56.067444 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.067385 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:56.075795 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.075768 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-openshift-service-ca\") pod \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " Oct 30 06:57:56.075989 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.075822 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-metrics-certs\") pod \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " Oct 30 06:57:56.075989 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.075884 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-kube-api-access-8bg5w\" (UniqueName: \"kubernetes.io/projected/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-kube-api-access-8bg5w\") pod \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " Oct 30 06:57:56.076538 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.076512 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-maas-api-tls\") pod \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\" (UID: \"e7b70aac-3036-44cb-ae26-d4e0bea0a45e\") " Oct 30 06:57:56.076847 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.076667 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-openshift-service-ca" pod "e7b70aac-3036-44cb-ae26-d4e0bea0a45e" (UID: "e7b70aac-3036-44cb-ae26-d4e0bea0a45e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:57:56.077716 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.077691 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:56.123271 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.115590 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-kube-api-access-8bg5w" pod "e7b70aac-3036-44cb-ae26-d4e0bea0a45e" (UID: "e7b70aac-3036-44cb-ae26-d4e0bea0a45e"). InnerVolumeSpecName "kube-api-access-8bg5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:57:56.132434 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.132392 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-metrics-certs" pod "e7b70aac-3036-44cb-ae26-d4e0bea0a45e" (UID: "e7b70aac-3036-44cb-ae26-d4e0bea0a45e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:56.132434 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.132410 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-maas-api-tls" pod "e7b70aac-3036-44cb-ae26-d4e0bea0a45e" (UID: "e7b70aac-3036-44cb-ae26-d4e0bea0a45e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:57:56.151881 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.151842 6999 generic.go:356] "Generic (PLEG): container finished" podID="e7b70aac-3036-44cb-ae26-d4e0bea0a45e" containerID="59b374922932d63b6fc83bf852d5f391d5512039c32af75ac387cf330cf471e6" exitCode=0 Oct 30 06:57:56.152078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.152002 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" event={"ID":"e7b70aac-3036-44cb-ae26-d4e0bea0a45e","Type":"ContainerDied","Data":"59b374922932d63b6fc83bf852d5f391d5512039c32af75ac387cf330cf471e6"} Oct 30 06:57:56.152078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.152054 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" event={"ID":"e7b70aac-3036-44cb-ae26-d4e0bea0a45e","Type":"ContainerDied","Data":"5219123d04b8dfc6bbbb86be0da8068e7ceb5ff97e669b6ffd3b1942dd36a253"} Oct 30 06:57:56.152203 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.152090 6999 scope.go:122] "RemoveContainer" containerID="59b374922932d63b6fc83bf852d5f391d5512039c32af75ac387cf330cf471e6" Oct 30 06:57:56.152279 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.152257 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp" Oct 30 06:57:56.176562 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.176549 6999 scope.go:122] "RemoveContainer" containerID="59b374922932d63b6fc83bf852d5f391d5512039c32af75ac387cf330cf471e6" Oct 30 06:57:56.176840 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:57:56.176817 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59b374922932d63b6fc83bf852d5f391d5512039c32af75ac387cf330cf471e6\": container with ID starting with 59b374922932d63b6fc83bf852d5f391d5512039c32af75ac387cf330cf471e6 not found: ID does not exist" containerID="59b374922932d63b6fc83bf852d5f391d5512039c32af75ac387cf330cf471e6" Oct 30 06:57:56.176919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.176848 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59b374922932d63b6fc83bf852d5f391d5512039c32af75ac387cf330cf471e6"} err="failed to get container status \"59b374922932d63b6fc83bf852d5f391d5512039c32af75ac387cf330cf471e6\": rpc error: code = NotFound desc = could not find container \"59b374922932d63b6fc83bf852d5f391d5512039c32af75ac387cf330cf471e6\": container with ID starting with 59b374922932d63b6fc83bf852d5f391d5512039c32af75ac387cf330cf471e6 not found: ID does not exist" Oct 30 06:57:56.178563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.178544 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:56.178631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.178571 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bg5w\" (UniqueName: \"kubernetes.io/projected/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-kube-api-access-8bg5w\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:56.178631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.178582 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7b70aac-3036-44cb-ae26-d4e0bea0a45e-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:57:56.207923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.207899 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp"] Oct 30 06:57:56.214721 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.214700 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8baf7e-8687c9d868-bnknp"] Oct 30 06:57:56.377758 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.377719 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:56.521559 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.521470 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:56.759044 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:56.759015 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:57.081111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:57.081082 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:57:57.613241 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:57.610043 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7b70aac-3036-44cb-ae26-d4e0bea0a45e" path="/var/lib/kubelet/pods/e7b70aac-3036-44cb-ae26-d4e0bea0a45e/volumes" Oct 30 06:57:58.692989 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:57:58.692956 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:00.263999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:00.263970 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:00.814680 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:00.814647 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:00.955874 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:00.955832 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:01.468023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.467986 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:01.470560 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.470538 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" podUID="79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" containerName="main" containerID="cri-o://c8a4a9702019f7ae10e7e131e4faf9f827811d093eb45e43b672d37e2a5c9f84" gracePeriod=45 Oct 30 06:58:01.739695 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.739666 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:58:01.847042 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.847011 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-home\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-home\") pod \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " Oct 30 06:58:01.847264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.847051 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-tmp-dir\") pod \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " Oct 30 06:58:01.847264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.847079 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-kube-api-access-rg66v\" (UniqueName: \"kubernetes.io/projected/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-kube-api-access-rg66v\") pod \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " Oct 30 06:58:01.847264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.847159 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-tls-certs\" (UniqueName: \"kubernetes.io/secret/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-tls-certs\") pod \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " Oct 30 06:58:01.847464 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.847303 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-dshm\") pod \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " Oct 30 06:58:01.847464 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.847386 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-model-cache\") pod \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\" (UID: \"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4\") " Oct 30 06:58:01.847464 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.847398 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-home" pod "79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" (UID: "79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.847464 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.847407 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-tmp-dir" pod "79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" (UID: "79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.847736 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.847711 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-model-cache" pod "79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" (UID: "79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.847973 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.847955 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.848041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.847980 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.848041 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.847994 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.849850 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.849822 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-tls-certs" pod "79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" (UID: "79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:01.850045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.850018 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-kube-api-access-rg66v" pod "79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" (UID: "79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4"). InnerVolumeSpecName "kube-api-access-rg66v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:01.850126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.850065 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-dshm" pod "79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" (UID: "79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:01.948893 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.948858 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.948893 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.948887 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:01.948893 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:01.948896 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg66v\" (UniqueName: \"kubernetes.io/projected/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4-kube-api-access-rg66v\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:02.197288 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:02.197246 6999 generic.go:356] "Generic (PLEG): container finished" podID="79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" containerID="c8a4a9702019f7ae10e7e131e4faf9f827811d093eb45e43b672d37e2a5c9f84" exitCode=0 Oct 30 06:58:02.197501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:02.197317 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" Oct 30 06:58:02.197501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:02.197342 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" event={"ID":"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4","Type":"ContainerDied","Data":"c8a4a9702019f7ae10e7e131e4faf9f827811d093eb45e43b672d37e2a5c9f84"} Oct 30 06:58:02.197501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:02.197400 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm" event={"ID":"79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4","Type":"ContainerDied","Data":"d42985b51bb9386f7079f03ef5a715db6a107bf8a22f613f3e00287ca1f69404"} Oct 30 06:58:02.197501 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:02.197424 6999 scope.go:122] "RemoveContainer" containerID="c8a4a9702019f7ae10e7e131e4faf9f827811d093eb45e43b672d37e2a5c9f84" Oct 30 06:58:02.209602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:02.209584 6999 scope.go:122] "RemoveContainer" containerID="c8a4a9702019f7ae10e7e131e4faf9f827811d093eb45e43b672d37e2a5c9f84" Oct 30 06:58:02.209936 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:58:02.209896 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a4a9702019f7ae10e7e131e4faf9f827811d093eb45e43b672d37e2a5c9f84\": container with ID starting with c8a4a9702019f7ae10e7e131e4faf9f827811d093eb45e43b672d37e2a5c9f84 not found: ID does not exist" containerID="c8a4a9702019f7ae10e7e131e4faf9f827811d093eb45e43b672d37e2a5c9f84" Oct 30 06:58:02.210046 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:02.209935 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a4a9702019f7ae10e7e131e4faf9f827811d093eb45e43b672d37e2a5c9f84"} err="failed to get container status \"c8a4a9702019f7ae10e7e131e4faf9f827811d093eb45e43b672d37e2a5c9f84\": rpc error: code = NotFound desc = could not find container \"c8a4a9702019f7ae10e7e131e4faf9f827811d093eb45e43b672d37e2a5c9f84\": container with ID starting with c8a4a9702019f7ae10e7e131e4faf9f827811d093eb45e43b672d37e2a5c9f84 not found: ID does not exist" Oct 30 06:58:02.249264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:02.249235 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm"] Oct 30 06:58:02.260940 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:02.260914 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a519e9/simulator-w0-a519e9-kserve-5898f479c-8wxtm"] Oct 30 06:58:02.724762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:02.724713 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:03.610665 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:03.610588 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" path="/var/lib/kubelet/pods/79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4/volumes" Oct 30 06:58:04.142603 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:58:04.142566 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:58:04.251007 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:04.250976 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:04.393446 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:04.393362 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:04.547805 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:04.547769 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:04.697305 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:04.697199 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:06.368693 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:06.368660 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:06.622695 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:06.622597 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:08.145005 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:08.144973 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:08.295181 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:08.295138 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:08.512066 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:08.511967 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:09.023066 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:09.023032 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:10.164355 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:10.164323 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:12.149818 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:12.149772 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:14.046265 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:14.046203 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:15.804991 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:15.804957 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:15.959770 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:15.959738 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:17.390796 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.390757 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n"] Oct 30 06:58:17.391351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.391099 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" podUID="0f526f75-f591-46d3-8bef-6d9c8b76ded6" containerName="maas-api" containerID="cri-o://d1fc4fdc53b2ec7436376a4a7a43277f69a4fa8462847a7b4cb380db7af38727" gracePeriod=30 Oct 30 06:58:17.482597 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.482562 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:17.711205 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.711169 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:58:17.837066 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.837030 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f526f75-f591-46d3-8bef-6d9c8b76ded6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f526f75-f591-46d3-8bef-6d9c8b76ded6-metrics-certs\") pod \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " Oct 30 06:58:17.837273 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.837254 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f526f75-f591-46d3-8bef-6d9c8b76ded6-kube-api-access-5pnfx\" (UniqueName: \"kubernetes.io/projected/0f526f75-f591-46d3-8bef-6d9c8b76ded6-kube-api-access-5pnfx\") pod \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " Oct 30 06:58:17.837363 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.837328 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f526f75-f591-46d3-8bef-6d9c8b76ded6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f526f75-f591-46d3-8bef-6d9c8b76ded6-maas-api-tls\") pod \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " Oct 30 06:58:17.837425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.837395 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f526f75-f591-46d3-8bef-6d9c8b76ded6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f526f75-f591-46d3-8bef-6d9c8b76ded6-openshift-service-ca\") pod \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\" (UID: \"0f526f75-f591-46d3-8bef-6d9c8b76ded6\") " Oct 30 06:58:17.838192 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.838161 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f526f75-f591-46d3-8bef-6d9c8b76ded6-openshift-service-ca" pod "0f526f75-f591-46d3-8bef-6d9c8b76ded6" (UID: "0f526f75-f591-46d3-8bef-6d9c8b76ded6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:17.839468 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.839443 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f526f75-f591-46d3-8bef-6d9c8b76ded6-metrics-certs" pod "0f526f75-f591-46d3-8bef-6d9c8b76ded6" (UID: "0f526f75-f591-46d3-8bef-6d9c8b76ded6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:17.839625 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.839609 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f526f75-f591-46d3-8bef-6d9c8b76ded6-kube-api-access-5pnfx" pod "0f526f75-f591-46d3-8bef-6d9c8b76ded6" (UID: "0f526f75-f591-46d3-8bef-6d9c8b76ded6"). InnerVolumeSpecName "kube-api-access-5pnfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:17.839952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.839928 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f526f75-f591-46d3-8bef-6d9c8b76ded6-maas-api-tls" pod "0f526f75-f591-46d3-8bef-6d9c8b76ded6" (UID: "0f526f75-f591-46d3-8bef-6d9c8b76ded6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:17.938968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.938883 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f526f75-f591-46d3-8bef-6d9c8b76ded6-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:17.938968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.938919 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f526f75-f591-46d3-8bef-6d9c8b76ded6-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:17.938968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.938929 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f526f75-f591-46d3-8bef-6d9c8b76ded6-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:17.938968 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:17.938938 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pnfx\" (UniqueName: \"kubernetes.io/projected/0f526f75-f591-46d3-8bef-6d9c8b76ded6-kube-api-access-5pnfx\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:18.033748 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:18.033708 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:18.295386 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:18.295295 6999 generic.go:356] "Generic (PLEG): container finished" podID="0f526f75-f591-46d3-8bef-6d9c8b76ded6" containerID="d1fc4fdc53b2ec7436376a4a7a43277f69a4fa8462847a7b4cb380db7af38727" exitCode=0 Oct 30 06:58:18.295386 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:18.295342 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" Oct 30 06:58:18.295654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:18.295398 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" event={"ID":"0f526f75-f591-46d3-8bef-6d9c8b76ded6","Type":"ContainerDied","Data":"d1fc4fdc53b2ec7436376a4a7a43277f69a4fa8462847a7b4cb380db7af38727"} Oct 30 06:58:18.295654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:18.295457 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n" event={"ID":"0f526f75-f591-46d3-8bef-6d9c8b76ded6","Type":"ContainerDied","Data":"584366f766622ac94141746147f5ccbf2a14b159bf906a9a6811ea87cd8bb04f"} Oct 30 06:58:18.295654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:18.295486 6999 scope.go:122] "RemoveContainer" containerID="d1fc4fdc53b2ec7436376a4a7a43277f69a4fa8462847a7b4cb380db7af38727" Oct 30 06:58:18.298349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:18.298323 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:18.306422 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:18.306403 6999 scope.go:122] "RemoveContainer" containerID="d1fc4fdc53b2ec7436376a4a7a43277f69a4fa8462847a7b4cb380db7af38727" Oct 30 06:58:18.306699 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:58:18.306676 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1fc4fdc53b2ec7436376a4a7a43277f69a4fa8462847a7b4cb380db7af38727\": container with ID starting with d1fc4fdc53b2ec7436376a4a7a43277f69a4fa8462847a7b4cb380db7af38727 not found: ID does not exist" containerID="d1fc4fdc53b2ec7436376a4a7a43277f69a4fa8462847a7b4cb380db7af38727" Oct 30 06:58:18.306756 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:18.306712 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1fc4fdc53b2ec7436376a4a7a43277f69a4fa8462847a7b4cb380db7af38727"} err="failed to get container status \"d1fc4fdc53b2ec7436376a4a7a43277f69a4fa8462847a7b4cb380db7af38727\": rpc error: code = NotFound desc = could not find container \"d1fc4fdc53b2ec7436376a4a7a43277f69a4fa8462847a7b4cb380db7af38727\": container with ID starting with d1fc4fdc53b2ec7436376a4a7a43277f69a4fa8462847a7b4cb380db7af38727 not found: ID does not exist" Oct 30 06:58:18.348204 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:18.348184 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n"] Oct 30 06:58:18.360108 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:18.360082 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-89f101-54797bddb7-c5h6n"] Oct 30 06:58:18.706682 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:18.706650 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:19.521741 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:19.521708 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8"] Oct 30 06:58:19.554833 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:19.554800 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns"] Oct 30 06:58:19.612013 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:19.611982 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f526f75-f591-46d3-8bef-6d9c8b76ded6" path="/var/lib/kubelet/pods/0f526f75-f591-46d3-8bef-6d9c8b76ded6/volumes" Oct 30 06:58:19.703257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:19.703223 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:19.847051 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:19.846964 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:20.019428 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:20.019394 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:20.308268 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:20.308203 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:21.719122 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:21.719096 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:23.472651 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:23.472618 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:24.513226 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:24.513154 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:24.522278 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:24.522245 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" podUID="e62ce8a0-4a42-4943-8ced-5f033661b9be" containerName="payload-processing" containerID="cri-o://039c75c6e16b0e87329ec47a45465a2953024ddc97b71ad7629a0d8d4bfb1564" gracePeriod=25 Oct 30 06:58:24.555238 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:24.555011 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" podUID="ded3226b-1e88-48ff-a599-268f411abc76" containerName="payload-pre-processing" containerID="cri-o://bd9315522559fabb00bac898d751a9a89f11244bfe8374534741219fb4997f70" gracePeriod=25 Oct 30 06:58:24.645381 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:24.645352 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:24.874258 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:24.874230 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" Oct 30 06:58:24.905137 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:24.905114 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" Oct 30 06:58:25.021662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.021624 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e62ce8a0-4a42-4943-8ced-5f033661b9be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62ce8a0-4a42-4943-8ced-5f033661b9be-plugins-config-volume\") pod \"e62ce8a0-4a42-4943-8ced-5f033661b9be\" (UID: \"e62ce8a0-4a42-4943-8ced-5f033661b9be\") " Oct 30 06:58:25.021865 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.021692 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e62ce8a0-4a42-4943-8ced-5f033661b9be-kube-api-access-zjnds\" (UniqueName: \"kubernetes.io/projected/e62ce8a0-4a42-4943-8ced-5f033661b9be-kube-api-access-zjnds\") pod \"e62ce8a0-4a42-4943-8ced-5f033661b9be\" (UID: \"e62ce8a0-4a42-4943-8ced-5f033661b9be\") " Oct 30 06:58:25.021865 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.021772 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded3226b-1e88-48ff-a599-268f411abc76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded3226b-1e88-48ff-a599-268f411abc76-plugins-config-volume\") pod \"ded3226b-1e88-48ff-a599-268f411abc76\" (UID: \"ded3226b-1e88-48ff-a599-268f411abc76\") " Oct 30 06:58:25.021999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.021893 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded3226b-1e88-48ff-a599-268f411abc76-kube-api-access-4z2xr\" (UniqueName: \"kubernetes.io/projected/ded3226b-1e88-48ff-a599-268f411abc76-kube-api-access-4z2xr\") pod \"ded3226b-1e88-48ff-a599-268f411abc76\" (UID: \"ded3226b-1e88-48ff-a599-268f411abc76\") " Oct 30 06:58:25.021999 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.021975 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e62ce8a0-4a42-4943-8ced-5f033661b9be-plugins-config-volume" pod "e62ce8a0-4a42-4943-8ced-5f033661b9be" (UID: "e62ce8a0-4a42-4943-8ced-5f033661b9be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:25.022199 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.022177 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded3226b-1e88-48ff-a599-268f411abc76-plugins-config-volume" pod "ded3226b-1e88-48ff-a599-268f411abc76" (UID: "ded3226b-1e88-48ff-a599-268f411abc76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:25.022356 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.022337 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ded3226b-1e88-48ff-a599-268f411abc76-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:25.022487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.022358 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e62ce8a0-4a42-4943-8ced-5f033661b9be-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:25.024061 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.024034 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62ce8a0-4a42-4943-8ced-5f033661b9be-kube-api-access-zjnds" pod "e62ce8a0-4a42-4943-8ced-5f033661b9be" (UID: "e62ce8a0-4a42-4943-8ced-5f033661b9be"). InnerVolumeSpecName "kube-api-access-zjnds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:25.024151 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.024104 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded3226b-1e88-48ff-a599-268f411abc76-kube-api-access-4z2xr" pod "ded3226b-1e88-48ff-a599-268f411abc76" (UID: "ded3226b-1e88-48ff-a599-268f411abc76"). InnerVolumeSpecName "kube-api-access-4z2xr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:25.123583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.123548 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z2xr\" (UniqueName: \"kubernetes.io/projected/ded3226b-1e88-48ff-a599-268f411abc76-kube-api-access-4z2xr\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:25.123583 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.123582 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjnds\" (UniqueName: \"kubernetes.io/projected/e62ce8a0-4a42-4943-8ced-5f033661b9be-kube-api-access-zjnds\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:25.335733 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.335690 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:25.339433 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.339403 6999 generic.go:356] "Generic (PLEG): container finished" podID="e62ce8a0-4a42-4943-8ced-5f033661b9be" containerID="039c75c6e16b0e87329ec47a45465a2953024ddc97b71ad7629a0d8d4bfb1564" exitCode=0 Oct 30 06:58:25.339611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.339460 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" Oct 30 06:58:25.339611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.339490 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" event={"ID":"e62ce8a0-4a42-4943-8ced-5f033661b9be","Type":"ContainerDied","Data":"039c75c6e16b0e87329ec47a45465a2953024ddc97b71ad7629a0d8d4bfb1564"} Oct 30 06:58:25.339611 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.339585 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8" event={"ID":"e62ce8a0-4a42-4943-8ced-5f033661b9be","Type":"ContainerDied","Data":"45761d82ffb46fb08a760c3febc41d18eede6e3a8da8b0b0eacfc45acbc58edf"} Oct 30 06:58:25.339824 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.339616 6999 scope.go:122] "RemoveContainer" containerID="039c75c6e16b0e87329ec47a45465a2953024ddc97b71ad7629a0d8d4bfb1564" Oct 30 06:58:25.341201 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.341176 6999 generic.go:356] "Generic (PLEG): container finished" podID="ded3226b-1e88-48ff-a599-268f411abc76" containerID="bd9315522559fabb00bac898d751a9a89f11244bfe8374534741219fb4997f70" exitCode=0 Oct 30 06:58:25.341519 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.341270 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" Oct 30 06:58:25.341655 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.341620 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" event={"ID":"ded3226b-1e88-48ff-a599-268f411abc76","Type":"ContainerDied","Data":"bd9315522559fabb00bac898d751a9a89f11244bfe8374534741219fb4997f70"} Oct 30 06:58:25.341783 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.341686 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns" event={"ID":"ded3226b-1e88-48ff-a599-268f411abc76","Type":"ContainerDied","Data":"98ddaeef3859ac5d8e11c140598d2b267b7fa393c18c3754ffb7f8526ca4e71b"} Oct 30 06:58:25.358448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.353710 6999 scope.go:122] "RemoveContainer" containerID="039c75c6e16b0e87329ec47a45465a2953024ddc97b71ad7629a0d8d4bfb1564" Oct 30 06:58:25.358448 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:58:25.353967 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"039c75c6e16b0e87329ec47a45465a2953024ddc97b71ad7629a0d8d4bfb1564\": container with ID starting with 039c75c6e16b0e87329ec47a45465a2953024ddc97b71ad7629a0d8d4bfb1564 not found: ID does not exist" containerID="039c75c6e16b0e87329ec47a45465a2953024ddc97b71ad7629a0d8d4bfb1564" Oct 30 06:58:25.358448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.354005 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"039c75c6e16b0e87329ec47a45465a2953024ddc97b71ad7629a0d8d4bfb1564"} err="failed to get container status \"039c75c6e16b0e87329ec47a45465a2953024ddc97b71ad7629a0d8d4bfb1564\": rpc error: code = NotFound desc = could not find container \"039c75c6e16b0e87329ec47a45465a2953024ddc97b71ad7629a0d8d4bfb1564\": container with ID starting with 039c75c6e16b0e87329ec47a45465a2953024ddc97b71ad7629a0d8d4bfb1564 not found: ID does not exist" Oct 30 06:58:25.358448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.354029 6999 scope.go:122] "RemoveContainer" containerID="bd9315522559fabb00bac898d751a9a89f11244bfe8374534741219fb4997f70" Oct 30 06:58:25.365339 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.365321 6999 scope.go:122] "RemoveContainer" containerID="bd9315522559fabb00bac898d751a9a89f11244bfe8374534741219fb4997f70" Oct 30 06:58:25.365620 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:58:25.365600 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd9315522559fabb00bac898d751a9a89f11244bfe8374534741219fb4997f70\": container with ID starting with bd9315522559fabb00bac898d751a9a89f11244bfe8374534741219fb4997f70 not found: ID does not exist" containerID="bd9315522559fabb00bac898d751a9a89f11244bfe8374534741219fb4997f70" Oct 30 06:58:25.365689 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.365627 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd9315522559fabb00bac898d751a9a89f11244bfe8374534741219fb4997f70"} err="failed to get container status \"bd9315522559fabb00bac898d751a9a89f11244bfe8374534741219fb4997f70\": rpc error: code = NotFound desc = could not find container \"bd9315522559fabb00bac898d751a9a89f11244bfe8374534741219fb4997f70\": container with ID starting with bd9315522559fabb00bac898d751a9a89f11244bfe8374534741219fb4997f70 not found: ID does not exist" Oct 30 06:58:25.412753 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.412618 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8"] Oct 30 06:58:25.417298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.417267 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-ad1661-7949885446-5w8j8"] Oct 30 06:58:25.457737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.457708 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns"] Oct 30 06:58:25.460905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.460874 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-ad1661-76fd7cf99-k9dns"] Oct 30 06:58:25.482427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.482401 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:25.609230 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.609183 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded3226b-1e88-48ff-a599-268f411abc76" path="/var/lib/kubelet/pods/ded3226b-1e88-48ff-a599-268f411abc76/volumes" Oct 30 06:58:25.609900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:25.609876 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e62ce8a0-4a42-4943-8ced-5f033661b9be" path="/var/lib/kubelet/pods/e62ce8a0-4a42-4943-8ced-5f033661b9be/volumes" Oct 30 06:58:26.261919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:26.261887 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:26.404707 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:26.404676 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:26.588499 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:26.588423 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:27.259389 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:27.259356 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:28.986070 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:28.986038 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:29.129879 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:29.129847 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:29.270749 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:29.270652 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:30.683786 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:30.683753 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:30.990468 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:30.990395 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb"] Oct 30 06:58:30.990688 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:30.990637 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" podUID="2e6794b5-28bc-4b10-a0d4-c1f529a09908" containerName="maas-api" containerID="cri-o://3b0cb8be02e8984fd68112bf0d371d59cb6be300a43aa189687d9eb1f2f86120" gracePeriod=30 Oct 30 06:58:31.085037 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.085001 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:31.225846 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.225174 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:31.315182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.315145 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:58:31.367163 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.367131 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:31.385979 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.385939 6999 generic.go:356] "Generic (PLEG): container finished" podID="2e6794b5-28bc-4b10-a0d4-c1f529a09908" containerID="3b0cb8be02e8984fd68112bf0d371d59cb6be300a43aa189687d9eb1f2f86120" exitCode=0 Oct 30 06:58:31.386175 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.386064 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" event={"ID":"2e6794b5-28bc-4b10-a0d4-c1f529a09908","Type":"ContainerDied","Data":"3b0cb8be02e8984fd68112bf0d371d59cb6be300a43aa189687d9eb1f2f86120"} Oct 30 06:58:31.386175 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.386115 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" event={"ID":"2e6794b5-28bc-4b10-a0d4-c1f529a09908","Type":"ContainerDied","Data":"8773a98c033f3c970278176fd44da7e4b101ed7806c6ffdcf7bd93b239de40c1"} Oct 30 06:58:31.386175 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.386146 6999 scope.go:122] "RemoveContainer" containerID="3b0cb8be02e8984fd68112bf0d371d59cb6be300a43aa189687d9eb1f2f86120" Oct 30 06:58:31.386377 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.386311 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb" Oct 30 06:58:31.426271 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.424311 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e6794b5-28bc-4b10-a0d4-c1f529a09908-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6794b5-28bc-4b10-a0d4-c1f529a09908-maas-api-tls\") pod \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " Oct 30 06:58:31.426271 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.424392 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e6794b5-28bc-4b10-a0d4-c1f529a09908-kube-api-access-4mq8g\" (UniqueName: \"kubernetes.io/projected/2e6794b5-28bc-4b10-a0d4-c1f529a09908-kube-api-access-4mq8g\") pod \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " Oct 30 06:58:31.426271 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.424429 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e6794b5-28bc-4b10-a0d4-c1f529a09908-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6794b5-28bc-4b10-a0d4-c1f529a09908-openshift-service-ca\") pod \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " Oct 30 06:58:31.426271 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.424479 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e6794b5-28bc-4b10-a0d4-c1f529a09908-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6794b5-28bc-4b10-a0d4-c1f529a09908-metrics-certs\") pod \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\" (UID: \"2e6794b5-28bc-4b10-a0d4-c1f529a09908\") " Oct 30 06:58:31.426271 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.424548 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb"] Oct 30 06:58:31.428727 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.428610 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6794b5-28bc-4b10-a0d4-c1f529a09908-metrics-certs" pod "2e6794b5-28bc-4b10-a0d4-c1f529a09908" (UID: "2e6794b5-28bc-4b10-a0d4-c1f529a09908"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:31.428978 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.428957 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6794b5-28bc-4b10-a0d4-c1f529a09908-openshift-service-ca" pod "2e6794b5-28bc-4b10-a0d4-c1f529a09908" (UID: "2e6794b5-28bc-4b10-a0d4-c1f529a09908"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:31.432463 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.432416 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6794b5-28bc-4b10-a0d4-c1f529a09908-maas-api-tls" pod "2e6794b5-28bc-4b10-a0d4-c1f529a09908" (UID: "2e6794b5-28bc-4b10-a0d4-c1f529a09908"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:31.435642 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.433994 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" podUID="a19ca2c4-c2ed-4d38-9537-fc394357aa84" containerName="maas-api" containerID="cri-o://62a393d0e22dae902c633b527f8f18928de16bf1a2587e7be158419d612a1889" gracePeriod=30 Oct 30 06:58:31.435911 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.435880 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6794b5-28bc-4b10-a0d4-c1f529a09908-kube-api-access-4mq8g" pod "2e6794b5-28bc-4b10-a0d4-c1f529a09908" (UID: "2e6794b5-28bc-4b10-a0d4-c1f529a09908"). InnerVolumeSpecName "kube-api-access-4mq8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:31.464726 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.463905 6999 scope.go:122] "RemoveContainer" containerID="3b0cb8be02e8984fd68112bf0d371d59cb6be300a43aa189687d9eb1f2f86120" Oct 30 06:58:31.464726 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:58:31.464473 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0cb8be02e8984fd68112bf0d371d59cb6be300a43aa189687d9eb1f2f86120\": container with ID starting with 3b0cb8be02e8984fd68112bf0d371d59cb6be300a43aa189687d9eb1f2f86120 not found: ID does not exist" containerID="3b0cb8be02e8984fd68112bf0d371d59cb6be300a43aa189687d9eb1f2f86120" Oct 30 06:58:31.464726 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.464502 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0cb8be02e8984fd68112bf0d371d59cb6be300a43aa189687d9eb1f2f86120"} err="failed to get container status \"3b0cb8be02e8984fd68112bf0d371d59cb6be300a43aa189687d9eb1f2f86120\": rpc error: code = NotFound desc = could not find container \"3b0cb8be02e8984fd68112bf0d371d59cb6be300a43aa189687d9eb1f2f86120\": container with ID starting with 3b0cb8be02e8984fd68112bf0d371d59cb6be300a43aa189687d9eb1f2f86120 not found: ID does not exist" Oct 30 06:58:31.513153 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.513073 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:31.526012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.525984 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e6794b5-28bc-4b10-a0d4-c1f529a09908-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.526012 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.526012 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mq8g\" (UniqueName: \"kubernetes.io/projected/2e6794b5-28bc-4b10-a0d4-c1f529a09908-kube-api-access-4mq8g\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.526172 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.526028 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e6794b5-28bc-4b10-a0d4-c1f529a09908-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.526172 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.526041 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e6794b5-28bc-4b10-a0d4-c1f529a09908-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.699350 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.699144 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:31.718626 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.718600 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:58:31.750580 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.750549 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb"] Oct 30 06:58:31.766787 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.766724 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-ad1661-6455879bb8-4c5zb"] Oct 30 06:58:31.829047 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.829007 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a19ca2c4-c2ed-4d38-9537-fc394357aa84-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19ca2c4-c2ed-4d38-9537-fc394357aa84-openshift-service-ca\") pod \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " Oct 30 06:58:31.829251 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.829182 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a19ca2c4-c2ed-4d38-9537-fc394357aa84-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19ca2c4-c2ed-4d38-9537-fc394357aa84-maas-api-tls\") pod \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " Oct 30 06:58:31.829337 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.829308 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a19ca2c4-c2ed-4d38-9537-fc394357aa84-kube-api-access-qkswd\" (UniqueName: \"kubernetes.io/projected/a19ca2c4-c2ed-4d38-9537-fc394357aa84-kube-api-access-qkswd\") pod \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " Oct 30 06:58:31.829427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.829361 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a19ca2c4-c2ed-4d38-9537-fc394357aa84-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a19ca2c4-c2ed-4d38-9537-fc394357aa84-metrics-certs\") pod \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\" (UID: \"a19ca2c4-c2ed-4d38-9537-fc394357aa84\") " Oct 30 06:58:31.829495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.829412 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a19ca2c4-c2ed-4d38-9537-fc394357aa84-openshift-service-ca" pod "a19ca2c4-c2ed-4d38-9537-fc394357aa84" (UID: "a19ca2c4-c2ed-4d38-9537-fc394357aa84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:31.829880 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.829856 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a19ca2c4-c2ed-4d38-9537-fc394357aa84-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.832245 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.832193 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19ca2c4-c2ed-4d38-9537-fc394357aa84-metrics-certs" pod "a19ca2c4-c2ed-4d38-9537-fc394357aa84" (UID: "a19ca2c4-c2ed-4d38-9537-fc394357aa84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:31.832331 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.832245 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a19ca2c4-c2ed-4d38-9537-fc394357aa84-maas-api-tls" pod "a19ca2c4-c2ed-4d38-9537-fc394357aa84" (UID: "a19ca2c4-c2ed-4d38-9537-fc394357aa84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:31.832331 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.832287 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19ca2c4-c2ed-4d38-9537-fc394357aa84-kube-api-access-qkswd" pod "a19ca2c4-c2ed-4d38-9537-fc394357aa84" (UID: "a19ca2c4-c2ed-4d38-9537-fc394357aa84"). InnerVolumeSpecName "kube-api-access-qkswd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:31.912785 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.912756 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:31.930571 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.930550 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a19ca2c4-c2ed-4d38-9537-fc394357aa84-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.930571 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.930570 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkswd\" (UniqueName: \"kubernetes.io/projected/a19ca2c4-c2ed-4d38-9537-fc394357aa84-kube-api-access-qkswd\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:31.930715 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:31.930579 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a19ca2c4-c2ed-4d38-9537-fc394357aa84-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:32.216050 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:32.216018 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:32.361318 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:32.361273 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:32.394803 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:32.394766 6999 generic.go:356] "Generic (PLEG): container finished" podID="a19ca2c4-c2ed-4d38-9537-fc394357aa84" containerID="62a393d0e22dae902c633b527f8f18928de16bf1a2587e7be158419d612a1889" exitCode=0 Oct 30 06:58:32.395010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:32.394880 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" event={"ID":"a19ca2c4-c2ed-4d38-9537-fc394357aa84","Type":"ContainerDied","Data":"62a393d0e22dae902c633b527f8f18928de16bf1a2587e7be158419d612a1889"} Oct 30 06:58:32.395010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:32.394935 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" event={"ID":"a19ca2c4-c2ed-4d38-9537-fc394357aa84","Type":"ContainerDied","Data":"9b42d057d17a3451798b51a8ca7d653e6e658bb9c0c3eca3e87f35195afac434"} Oct 30 06:58:32.395010 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:32.394960 6999 scope.go:122] "RemoveContainer" containerID="62a393d0e22dae902c633b527f8f18928de16bf1a2587e7be158419d612a1889" Oct 30 06:58:32.395244 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:32.395046 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb" Oct 30 06:58:32.406028 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:32.406008 6999 scope.go:122] "RemoveContainer" containerID="62a393d0e22dae902c633b527f8f18928de16bf1a2587e7be158419d612a1889" Oct 30 06:58:32.406345 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:58:32.406324 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a393d0e22dae902c633b527f8f18928de16bf1a2587e7be158419d612a1889\": container with ID starting with 62a393d0e22dae902c633b527f8f18928de16bf1a2587e7be158419d612a1889 not found: ID does not exist" containerID="62a393d0e22dae902c633b527f8f18928de16bf1a2587e7be158419d612a1889" Oct 30 06:58:32.406417 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:32.406354 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a393d0e22dae902c633b527f8f18928de16bf1a2587e7be158419d612a1889"} err="failed to get container status \"62a393d0e22dae902c633b527f8f18928de16bf1a2587e7be158419d612a1889\": rpc error: code = NotFound desc = could not find container \"62a393d0e22dae902c633b527f8f18928de16bf1a2587e7be158419d612a1889\": container with ID starting with 62a393d0e22dae902c633b527f8f18928de16bf1a2587e7be158419d612a1889 not found: ID does not exist" Oct 30 06:58:32.446899 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:32.446873 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb"] Oct 30 06:58:32.452861 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:32.452839 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-be48b4-765b9b8f4f-wq4jb"] Oct 30 06:58:32.507322 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:32.507271 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:32.863666 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:32.863634 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:33.608773 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:33.608699 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e6794b5-28bc-4b10-a0d4-c1f529a09908" path="/var/lib/kubelet/pods/2e6794b5-28bc-4b10-a0d4-c1f529a09908/volumes" Oct 30 06:58:33.609160 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:33.609145 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a19ca2c4-c2ed-4d38-9537-fc394357aa84" path="/var/lib/kubelet/pods/a19ca2c4-c2ed-4d38-9537-fc394357aa84/volumes" Oct 30 06:58:34.142712 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:58:34.142671 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:58:34.291150 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:34.291110 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:36.079989 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:36.079950 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:37.890790 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:37.890736 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:38.034764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:38.034733 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:39.922125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:39.922090 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:40.067717 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:40.067669 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:40.200826 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:40.200758 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:40.275920 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:40.275882 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g"] Oct 30 06:58:40.276167 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:40.276140 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" podUID="f85d5059-e45f-418d-ae2c-2712a3241086" containerName="istio-proxy" containerID="cri-o://6d5a063896c180814f634680a60853b68224317f34acb1e455a4c881b16b0fa3" gracePeriod=30 Oct 30 06:58:40.337849 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:40.337815 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:40.396739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:40.396705 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk"] Oct 30 06:58:40.396996 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:40.396968 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" podUID="b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" containerName="istio-proxy" containerID="cri-o://0c7a17ce0ce13651dcbca2e4787d9df892813081fb7b7d5e0c75588f17417cbf" gracePeriod=30 Oct 30 06:58:40.475762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:40.475687 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:40.668623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:40.668589 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:41.747628 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:41.747598 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:43.640073 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:43.640042 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:45.473400 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.473368 6999 generic.go:356] "Generic (PLEG): container finished" podID="f85d5059-e45f-418d-ae2c-2712a3241086" containerID="6d5a063896c180814f634680a60853b68224317f34acb1e455a4c881b16b0fa3" exitCode=0 Oct 30 06:58:45.474007 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.473444 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" event={"ID":"f85d5059-e45f-418d-ae2c-2712a3241086","Type":"ContainerDied","Data":"6d5a063896c180814f634680a60853b68224317f34acb1e455a4c881b16b0fa3"} Oct 30 06:58:45.533459 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.533398 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:45.542476 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.542419 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:58:45.677868 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.677846 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:58:45.694832 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.694806 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-workload-socket\") pod \"f85d5059-e45f-418d-ae2c-2712a3241086\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " Oct 30 06:58:45.695006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.694877 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f85d5059-e45f-418d-ae2c-2712a3241086-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85d5059-e45f-418d-ae2c-2712a3241086-istiod-ca-cert\") pod \"f85d5059-e45f-418d-ae2c-2712a3241086\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " Oct 30 06:58:45.695006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.694935 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-istio-envoy\") pod \"f85d5059-e45f-418d-ae2c-2712a3241086\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " Oct 30 06:58:45.695006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.694975 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f85d5059-e45f-418d-ae2c-2712a3241086-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85d5059-e45f-418d-ae2c-2712a3241086-istio-podinfo\") pod \"f85d5059-e45f-418d-ae2c-2712a3241086\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " Oct 30 06:58:45.695006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.695011 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85d5059-e45f-418d-ae2c-2712a3241086-istio-token\" (UniqueName: \"kubernetes.io/projected/f85d5059-e45f-418d-ae2c-2712a3241086-istio-token\") pod \"f85d5059-e45f-418d-ae2c-2712a3241086\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " Oct 30 06:58:45.695006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.695036 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85d5059-e45f-418d-ae2c-2712a3241086-kube-api-access-v22r4\" (UniqueName: \"kubernetes.io/projected/f85d5059-e45f-418d-ae2c-2712a3241086-kube-api-access-v22r4\") pod \"f85d5059-e45f-418d-ae2c-2712a3241086\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " Oct 30 06:58:45.695006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.695127 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-credential-socket\") pod \"f85d5059-e45f-418d-ae2c-2712a3241086\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " Oct 30 06:58:45.695006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.695169 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-workload-socket" pod "f85d5059-e45f-418d-ae2c-2712a3241086" (UID: "f85d5059-e45f-418d-ae2c-2712a3241086"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:45.695870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.695193 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-istio-data\") pod \"f85d5059-e45f-418d-ae2c-2712a3241086\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " Oct 30 06:58:45.695870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.695302 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-workload-certs\") pod \"f85d5059-e45f-418d-ae2c-2712a3241086\" (UID: \"f85d5059-e45f-418d-ae2c-2712a3241086\") " Oct 30 06:58:45.695870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.695380 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f85d5059-e45f-418d-ae2c-2712a3241086-istiod-ca-cert" pod "f85d5059-e45f-418d-ae2c-2712a3241086" (UID: "f85d5059-e45f-418d-ae2c-2712a3241086"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:45.695870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.695761 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-istio-data" pod "f85d5059-e45f-418d-ae2c-2712a3241086" (UID: "f85d5059-e45f-418d-ae2c-2712a3241086"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:45.696106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.695985 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-workload-certs" pod "f85d5059-e45f-418d-ae2c-2712a3241086" (UID: "f85d5059-e45f-418d-ae2c-2712a3241086"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:45.696106 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.696010 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-credential-socket" pod "f85d5059-e45f-418d-ae2c-2712a3241086" (UID: "f85d5059-e45f-418d-ae2c-2712a3241086"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:45.696257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.696228 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.696257 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.696251 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.696369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.696266 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.696369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.696279 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.696369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.696293 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f85d5059-e45f-418d-ae2c-2712a3241086-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.697926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.697901 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f85d5059-e45f-418d-ae2c-2712a3241086-istio-podinfo" pod "f85d5059-e45f-418d-ae2c-2712a3241086" (UID: "f85d5059-e45f-418d-ae2c-2712a3241086"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:58:45.697926 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.697909 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85d5059-e45f-418d-ae2c-2712a3241086-istio-token" pod "f85d5059-e45f-418d-ae2c-2712a3241086" (UID: "f85d5059-e45f-418d-ae2c-2712a3241086"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:45.698071 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.698039 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-istio-envoy" pod "f85d5059-e45f-418d-ae2c-2712a3241086" (UID: "f85d5059-e45f-418d-ae2c-2712a3241086"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:45.698124 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.698106 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85d5059-e45f-418d-ae2c-2712a3241086-kube-api-access-v22r4" pod "f85d5059-e45f-418d-ae2c-2712a3241086" (UID: "f85d5059-e45f-418d-ae2c-2712a3241086"). InnerVolumeSpecName "kube-api-access-v22r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:45.796934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.796860 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-envoy\") pod \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " Oct 30 06:58:45.796934 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.796900 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-podinfo\") pod \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " Oct 30 06:58:45.797296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.796937 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-credential-socket\") pod \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " Oct 30 06:58:45.797296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.797018 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-token\" (UniqueName: \"kubernetes.io/projected/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-token\") pod \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " Oct 30 06:58:45.797296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.797076 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-workload-certs\") pod \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " Oct 30 06:58:45.797296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.797180 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istiod-ca-cert\") pod \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " Oct 30 06:58:45.797296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.797265 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-workload-socket\") pod \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " Oct 30 06:58:45.797296 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.797259 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-credential-socket" pod "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" (UID: "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:45.797654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.797330 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-kube-api-access-sjc2w\" (UniqueName: \"kubernetes.io/projected/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-kube-api-access-sjc2w\") pod \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " Oct 30 06:58:45.797654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.797368 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-workload-certs" pod "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" (UID: "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:45.797654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.797383 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-data\") pod \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\" (UID: \"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9\") " Oct 30 06:58:45.797654 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.797515 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-workload-socket" pod "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" (UID: "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:45.797852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.797809 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-data" pod "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" (UID: "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:45.797852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.797777 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istiod-ca-cert" pod "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" (UID: "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:45.797956 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.797939 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.798008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.797965 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.798008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.797981 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f85d5059-e45f-418d-ae2c-2712a3241086-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.798008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.797996 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.798126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.798009 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f85d5059-e45f-418d-ae2c-2712a3241086-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.798126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.798022 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f85d5059-e45f-418d-ae2c-2712a3241086-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.798126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.798037 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v22r4\" (UniqueName: \"kubernetes.io/projected/f85d5059-e45f-418d-ae2c-2712a3241086-kube-api-access-v22r4\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.798126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.798051 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.798126 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.798067 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.799435 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.799397 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-envoy" pod "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" (UID: "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:58:45.799548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.799494 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-token" pod "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" (UID: "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:45.800001 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.799966 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-podinfo" pod "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" (UID: "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:58:45.800099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.800003 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-kube-api-access-sjc2w" pod "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" (UID: "b30fe423-3a43-4f07-aa4a-f147ea8f7cd9"). InnerVolumeSpecName "kube-api-access-sjc2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:45.899029 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.898999 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjc2w\" (UniqueName: \"kubernetes.io/projected/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-kube-api-access-sjc2w\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.899029 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.899028 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.899029 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.899038 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:45.899310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:45.899046 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:46.481006 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:46.480975 6999 generic.go:356] "Generic (PLEG): container finished" podID="b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" containerID="0c7a17ce0ce13651dcbca2e4787d9df892813081fb7b7d5e0c75588f17417cbf" exitCode=0 Oct 30 06:58:46.481493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:46.481022 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" Oct 30 06:58:46.481493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:46.481065 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" event={"ID":"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9","Type":"ContainerDied","Data":"0c7a17ce0ce13651dcbca2e4787d9df892813081fb7b7d5e0c75588f17417cbf"} Oct 30 06:58:46.481493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:46.481122 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk" event={"ID":"b30fe423-3a43-4f07-aa4a-f147ea8f7cd9","Type":"ContainerDied","Data":"c02a9a3776876b94238dcc80ed15ca8b0432b40ceb30209f619c27f0b123d38f"} Oct 30 06:58:46.481493 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:46.481149 6999 scope.go:122] "RemoveContainer" containerID="0c7a17ce0ce13651dcbca2e4787d9df892813081fb7b7d5e0c75588f17417cbf" Oct 30 06:58:46.482941 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:46.482923 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" Oct 30 06:58:46.483052 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:46.482935 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g" event={"ID":"f85d5059-e45f-418d-ae2c-2712a3241086","Type":"ContainerDied","Data":"4eef34ab355fcc5625c72e22ed82b089cbc3cc8c7ff8598d22d3b9b46f39c109"} Oct 30 06:58:46.494103 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:46.494086 6999 scope.go:122] "RemoveContainer" containerID="0c7a17ce0ce13651dcbca2e4787d9df892813081fb7b7d5e0c75588f17417cbf" Oct 30 06:58:46.494438 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:58:46.494415 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c7a17ce0ce13651dcbca2e4787d9df892813081fb7b7d5e0c75588f17417cbf\": container with ID starting with 0c7a17ce0ce13651dcbca2e4787d9df892813081fb7b7d5e0c75588f17417cbf not found: ID does not exist" containerID="0c7a17ce0ce13651dcbca2e4787d9df892813081fb7b7d5e0c75588f17417cbf" Oct 30 06:58:46.494508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:46.494446 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7a17ce0ce13651dcbca2e4787d9df892813081fb7b7d5e0c75588f17417cbf"} err="failed to get container status \"0c7a17ce0ce13651dcbca2e4787d9df892813081fb7b7d5e0c75588f17417cbf\": rpc error: code = NotFound desc = could not find container \"0c7a17ce0ce13651dcbca2e4787d9df892813081fb7b7d5e0c75588f17417cbf\": container with ID starting with 0c7a17ce0ce13651dcbca2e4787d9df892813081fb7b7d5e0c75588f17417cbf not found: ID does not exist" Oct 30 06:58:46.494508 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:46.494464 6999 scope.go:122] "RemoveContainer" containerID="6d5a063896c180814f634680a60853b68224317f34acb1e455a4c881b16b0fa3" Oct 30 06:58:46.540596 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:46.540576 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g"] Oct 30 06:58:46.552661 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:46.552643 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-ad1661-openshift-default-7fdc5689d9-cpw7g"] Oct 30 06:58:46.578570 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:46.578518 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk"] Oct 30 06:58:46.582328 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:46.582309 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a519e9-openshift-default-b7b85d565-dzrnk"] Oct 30 06:58:47.371008 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:47.370977 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:47.608128 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:47.608087 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" path="/var/lib/kubelet/pods/b30fe423-3a43-4f07-aa4a-f147ea8f7cd9/volumes" Oct 30 06:58:47.608635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:47.608610 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f85d5059-e45f-418d-ae2c-2712a3241086" path="/var/lib/kubelet/pods/f85d5059-e45f-418d-ae2c-2712a3241086/volumes" Oct 30 06:58:47.754371 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:47.754272 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:49.126702 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:49.126673 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:50.912703 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:50.912673 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:51.913613 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:51.913586 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:52.050502 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.050471 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:52.367952 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.367898 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb"] Oct 30 06:58:52.368168 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.368140 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" podUID="8111e1b2-a50e-4d15-8efb-00febd24b5e3" containerName="maas-api" containerID="cri-o://be19481bf527ae593c08c3911eaabc0eb3a0bace69fb39930da0ff18115ba738" gracePeriod=30 Oct 30 06:58:52.457840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.457811 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:52.534849 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.534815 6999 generic.go:356] "Generic (PLEG): container finished" podID="8111e1b2-a50e-4d15-8efb-00febd24b5e3" containerID="be19481bf527ae593c08c3911eaabc0eb3a0bace69fb39930da0ff18115ba738" exitCode=0 Oct 30 06:58:52.535040 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.534886 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" event={"ID":"8111e1b2-a50e-4d15-8efb-00febd24b5e3","Type":"ContainerDied","Data":"be19481bf527ae593c08c3911eaabc0eb3a0bace69fb39930da0ff18115ba738"} Oct 30 06:58:52.650883 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.650852 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:52.671929 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.671900 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:58:52.769149 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.769119 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8111e1b2-a50e-4d15-8efb-00febd24b5e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8111e1b2-a50e-4d15-8efb-00febd24b5e3-metrics-certs\") pod \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " Oct 30 06:58:52.769380 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.769169 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8111e1b2-a50e-4d15-8efb-00febd24b5e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8111e1b2-a50e-4d15-8efb-00febd24b5e3-maas-api-tls\") pod \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " Oct 30 06:58:52.771510 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.771482 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8111e1b2-a50e-4d15-8efb-00febd24b5e3-metrics-certs" pod "8111e1b2-a50e-4d15-8efb-00febd24b5e3" (UID: "8111e1b2-a50e-4d15-8efb-00febd24b5e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:52.771657 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.771578 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8111e1b2-a50e-4d15-8efb-00febd24b5e3-maas-api-tls" pod "8111e1b2-a50e-4d15-8efb-00febd24b5e3" (UID: "8111e1b2-a50e-4d15-8efb-00febd24b5e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:52.793866 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.793830 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:52.870335 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.870306 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8111e1b2-a50e-4d15-8efb-00febd24b5e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8111e1b2-a50e-4d15-8efb-00febd24b5e3-openshift-service-ca\") pod \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " Oct 30 06:58:52.870533 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.870367 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8111e1b2-a50e-4d15-8efb-00febd24b5e3-kube-api-access-p25hn\" (UniqueName: \"kubernetes.io/projected/8111e1b2-a50e-4d15-8efb-00febd24b5e3-kube-api-access-p25hn\") pod \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\" (UID: \"8111e1b2-a50e-4d15-8efb-00febd24b5e3\") " Oct 30 06:58:52.870719 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.870696 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8111e1b2-a50e-4d15-8efb-00febd24b5e3-openshift-service-ca" pod "8111e1b2-a50e-4d15-8efb-00febd24b5e3" (UID: "8111e1b2-a50e-4d15-8efb-00febd24b5e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:52.870799 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.870783 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8111e1b2-a50e-4d15-8efb-00febd24b5e3-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:52.870852 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.870799 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8111e1b2-a50e-4d15-8efb-00febd24b5e3-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:52.874779 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.874745 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8111e1b2-a50e-4d15-8efb-00febd24b5e3-kube-api-access-p25hn" pod "8111e1b2-a50e-4d15-8efb-00febd24b5e3" (UID: "8111e1b2-a50e-4d15-8efb-00febd24b5e3"). InnerVolumeSpecName "kube-api-access-p25hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:52.971572 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.971493 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8111e1b2-a50e-4d15-8efb-00febd24b5e3-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:52.971572 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:52.971519 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p25hn\" (UniqueName: \"kubernetes.io/projected/8111e1b2-a50e-4d15-8efb-00febd24b5e3-kube-api-access-p25hn\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:53.004635 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:53.004602 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:53.154269 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:53.154137 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:53.542124 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:53.542099 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" Oct 30 06:58:53.542319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:53.542185 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb" event={"ID":"8111e1b2-a50e-4d15-8efb-00febd24b5e3","Type":"ContainerDied","Data":"a81d1d4dda16a8f67325d03dadb559759c160d9f22717d667d58e011468b2d50"} Oct 30 06:58:53.542319 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:53.542276 6999 scope.go:122] "RemoveContainer" containerID="be19481bf527ae593c08c3911eaabc0eb3a0bace69fb39930da0ff18115ba738" Oct 30 06:58:53.580074 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:58:53.580052 6999 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8111e1b2_a50e_4d15_8efb_00febd24b5e3.slice/crio-a81d1d4dda16a8f67325d03dadb559759c160d9f22717d667d58e011468b2d50\": RecentStats: unable to find data in memory cache]" Oct 30 06:58:53.595977 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:53.595956 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb"] Oct 30 06:58:53.608629 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:53.608607 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-b8ca49-84b678bc74-xswrb"] Oct 30 06:58:53.683142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:53.683082 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:54.553166 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.553132 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg"] Oct 30 06:58:54.553712 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.553385 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" podUID="1e88bddc-e0c1-4516-bd47-356b38380639" containerName="maas-api" containerID="cri-o://a8f5709f69eef6bb720308de728d176efe2f2380500eca8c121bd04a43c349ef" gracePeriod=30 Oct 30 06:58:54.649833 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.649788 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:54.810840 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.810764 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:58:54.891471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.891435 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e88bddc-e0c1-4516-bd47-356b38380639-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e88bddc-e0c1-4516-bd47-356b38380639-openshift-service-ca\") pod \"1e88bddc-e0c1-4516-bd47-356b38380639\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " Oct 30 06:58:54.891671 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.891552 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-maas-api-tls\") pod \"1e88bddc-e0c1-4516-bd47-356b38380639\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " Oct 30 06:58:54.891671 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.891586 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-metrics-certs\") pod \"1e88bddc-e0c1-4516-bd47-356b38380639\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " Oct 30 06:58:54.891815 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.891680 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e88bddc-e0c1-4516-bd47-356b38380639-kube-api-access-qfqmk\" (UniqueName: \"kubernetes.io/projected/1e88bddc-e0c1-4516-bd47-356b38380639-kube-api-access-qfqmk\") pod \"1e88bddc-e0c1-4516-bd47-356b38380639\" (UID: \"1e88bddc-e0c1-4516-bd47-356b38380639\") " Oct 30 06:58:54.891886 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.891852 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e88bddc-e0c1-4516-bd47-356b38380639-openshift-service-ca" pod "1e88bddc-e0c1-4516-bd47-356b38380639" (UID: "1e88bddc-e0c1-4516-bd47-356b38380639"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:58:54.892176 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.892153 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e88bddc-e0c1-4516-bd47-356b38380639-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:54.894024 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.893996 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-maas-api-tls" pod "1e88bddc-e0c1-4516-bd47-356b38380639" (UID: "1e88bddc-e0c1-4516-bd47-356b38380639"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:54.894394 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.894375 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e88bddc-e0c1-4516-bd47-356b38380639-kube-api-access-qfqmk" pod "1e88bddc-e0c1-4516-bd47-356b38380639" (UID: "1e88bddc-e0c1-4516-bd47-356b38380639"). InnerVolumeSpecName "kube-api-access-qfqmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:58:54.894505 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.894413 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-metrics-certs" pod "1e88bddc-e0c1-4516-bd47-356b38380639" (UID: "1e88bddc-e0c1-4516-bd47-356b38380639"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:58:54.993482 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.993445 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfqmk\" (UniqueName: \"kubernetes.io/projected/1e88bddc-e0c1-4516-bd47-356b38380639-kube-api-access-qfqmk\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:54.993482 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.993485 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:54.993688 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:54.993501 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e88bddc-e0c1-4516-bd47-356b38380639-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:58:55.203602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:55.203571 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:55.359307 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:55.359274 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:55.555412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:55.555315 6999 generic.go:356] "Generic (PLEG): container finished" podID="1e88bddc-e0c1-4516-bd47-356b38380639" containerID="a8f5709f69eef6bb720308de728d176efe2f2380500eca8c121bd04a43c349ef" exitCode=0 Oct 30 06:58:55.555412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:55.555374 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" Oct 30 06:58:55.555412 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:55.555383 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" event={"ID":"1e88bddc-e0c1-4516-bd47-356b38380639","Type":"ContainerDied","Data":"a8f5709f69eef6bb720308de728d176efe2f2380500eca8c121bd04a43c349ef"} Oct 30 06:58:55.556032 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:55.555454 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg" event={"ID":"1e88bddc-e0c1-4516-bd47-356b38380639","Type":"ContainerDied","Data":"5d035b847301240ba435734c94d860d5c55b5cb6e8164db56d92bc88506390f3"} Oct 30 06:58:55.556032 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:55.555486 6999 scope.go:122] "RemoveContainer" containerID="a8f5709f69eef6bb720308de728d176efe2f2380500eca8c121bd04a43c349ef" Oct 30 06:58:55.566407 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:55.566390 6999 scope.go:122] "RemoveContainer" containerID="a8f5709f69eef6bb720308de728d176efe2f2380500eca8c121bd04a43c349ef" Oct 30 06:58:55.566680 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:58:55.566661 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f5709f69eef6bb720308de728d176efe2f2380500eca8c121bd04a43c349ef\": container with ID starting with a8f5709f69eef6bb720308de728d176efe2f2380500eca8c121bd04a43c349ef not found: ID does not exist" containerID="a8f5709f69eef6bb720308de728d176efe2f2380500eca8c121bd04a43c349ef" Oct 30 06:58:55.566768 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:55.566686 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f5709f69eef6bb720308de728d176efe2f2380500eca8c121bd04a43c349ef"} err="failed to get container status \"a8f5709f69eef6bb720308de728d176efe2f2380500eca8c121bd04a43c349ef\": rpc error: code = NotFound desc = could not find container \"a8f5709f69eef6bb720308de728d176efe2f2380500eca8c121bd04a43c349ef\": container with ID starting with a8f5709f69eef6bb720308de728d176efe2f2380500eca8c121bd04a43c349ef not found: ID does not exist" Oct 30 06:58:55.608655 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:55.608634 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8111e1b2-a50e-4d15-8efb-00febd24b5e3" path="/var/lib/kubelet/pods/8111e1b2-a50e-4d15-8efb-00febd24b5e3/volumes" Oct 30 06:58:55.608923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:55.608910 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg"] Oct 30 06:58:55.618638 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:55.618619 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-f0145f-9cb555967-lpxwg"] Oct 30 06:58:55.846772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:55.846694 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:57.327643 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:57.327611 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:58:57.609503 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:57.609475 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e88bddc-e0c1-4516-bd47-356b38380639" path="/var/lib/kubelet/pods/1e88bddc-e0c1-4516-bd47-356b38380639/volumes" Oct 30 06:58:59.029282 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:58:59.029245 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:00.345815 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:00.345778 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:00.489310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:00.489280 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:00.697624 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:00.697595 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:00.871390 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:00.871358 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:02.759794 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:02.759758 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:03.231444 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:03.231408 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:03.375837 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:03.375808 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:03.608532 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:03.608509 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:04.143383 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:59:04.143345 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:59:04.398916 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.398823 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq"] Oct 30 06:59:04.399097 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.399038 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" podUID="4c821205-038e-4dd0-b122-837935d53312" containerName="maas-api" containerID="cri-o://ac82548fe44476e02eee78ce29ab10e029b53ccb57e0b09ae6361787a0bb56e8" gracePeriod=30 Oct 30 06:59:04.509283 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.509234 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:04.526925 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.526879 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf"] Oct 30 06:59:04.527196 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.527148 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" podUID="5f2aec32-15b0-4a59-839b-a1e23f337319" containerName="istio-proxy" containerID="cri-o://e61e34e3f27e7678025586f49f31b8126dbb86cca558b26710eafefb34b0f130" gracePeriod=30 Oct 30 06:59:04.623555 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.623519 6999 generic.go:356] "Generic (PLEG): container finished" podID="4c821205-038e-4dd0-b122-837935d53312" containerID="ac82548fe44476e02eee78ce29ab10e029b53ccb57e0b09ae6361787a0bb56e8" exitCode=0 Oct 30 06:59:04.623721 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.623612 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" event={"ID":"4c821205-038e-4dd0-b122-837935d53312","Type":"ContainerDied","Data":"ac82548fe44476e02eee78ce29ab10e029b53ccb57e0b09ae6361787a0bb56e8"} Oct 30 06:59:04.722823 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.722795 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:59:04.804789 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.804752 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c821205-038e-4dd0-b122-837935d53312-kube-api-access-6wrb5\" (UniqueName: \"kubernetes.io/projected/4c821205-038e-4dd0-b122-837935d53312-kube-api-access-6wrb5\") pod \"4c821205-038e-4dd0-b122-837935d53312\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " Oct 30 06:59:04.804993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.804850 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c821205-038e-4dd0-b122-837935d53312-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c821205-038e-4dd0-b122-837935d53312-openshift-service-ca\") pod \"4c821205-038e-4dd0-b122-837935d53312\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " Oct 30 06:59:04.804993 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.804948 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c821205-038e-4dd0-b122-837935d53312-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c821205-038e-4dd0-b122-837935d53312-metrics-certs\") pod \"4c821205-038e-4dd0-b122-837935d53312\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " Oct 30 06:59:04.805136 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.805008 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c821205-038e-4dd0-b122-837935d53312-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c821205-038e-4dd0-b122-837935d53312-maas-api-tls\") pod \"4c821205-038e-4dd0-b122-837935d53312\" (UID: \"4c821205-038e-4dd0-b122-837935d53312\") " Oct 30 06:59:04.805656 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.805627 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c821205-038e-4dd0-b122-837935d53312-openshift-service-ca" pod "4c821205-038e-4dd0-b122-837935d53312" (UID: "4c821205-038e-4dd0-b122-837935d53312"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:04.807820 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.807758 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c821205-038e-4dd0-b122-837935d53312-metrics-certs" pod "4c821205-038e-4dd0-b122-837935d53312" (UID: "4c821205-038e-4dd0-b122-837935d53312"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:04.807950 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.807792 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c821205-038e-4dd0-b122-837935d53312-kube-api-access-6wrb5" pod "4c821205-038e-4dd0-b122-837935d53312" (UID: "4c821205-038e-4dd0-b122-837935d53312"). InnerVolumeSpecName "kube-api-access-6wrb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:04.807950 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.807794 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c821205-038e-4dd0-b122-837935d53312-maas-api-tls" pod "4c821205-038e-4dd0-b122-837935d53312" (UID: "4c821205-038e-4dd0-b122-837935d53312"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:04.906762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.906712 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wrb5\" (UniqueName: \"kubernetes.io/projected/4c821205-038e-4dd0-b122-837935d53312-kube-api-access-6wrb5\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:04.906762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.906755 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c821205-038e-4dd0-b122-837935d53312-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:04.906762 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.906770 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c821205-038e-4dd0-b122-837935d53312-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:04.907168 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:04.906784 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c821205-038e-4dd0-b122-837935d53312-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:05.158581 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:05.158553 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:05.301349 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:05.301321 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:05.630057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:05.630033 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" Oct 30 06:59:05.630195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:05.630070 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq" event={"ID":"4c821205-038e-4dd0-b122-837935d53312","Type":"ContainerDied","Data":"76570b166770f1f41b4a21143b73ec4d32925461734ba6959f2c8b160e27b24a"} Oct 30 06:59:05.630195 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:05.630132 6999 scope.go:122] "RemoveContainer" containerID="ac82548fe44476e02eee78ce29ab10e029b53ccb57e0b09ae6361787a0bb56e8" Oct 30 06:59:05.681136 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:05.681104 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq"] Oct 30 06:59:05.693446 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:05.693415 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6d1778-5449dd4d5-8mfpq"] Oct 30 06:59:05.810584 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:05.810554 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:06.677696 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:06.677661 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:07.229973 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:07.229938 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:07.608375 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:07.608350 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c821205-038e-4dd0-b122-837935d53312" path="/var/lib/kubelet/pods/4c821205-038e-4dd0-b122-837935d53312/volumes" Oct 30 06:59:08.976525 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:08.976478 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:09.172510 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.172473 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:09.654142 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.654113 6999 generic.go:356] "Generic (PLEG): container finished" podID="5f2aec32-15b0-4a59-839b-a1e23f337319" containerID="e61e34e3f27e7678025586f49f31b8126dbb86cca558b26710eafefb34b0f130" exitCode=0 Oct 30 06:59:09.654285 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.654182 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" event={"ID":"5f2aec32-15b0-4a59-839b-a1e23f337319","Type":"ContainerDied","Data":"e61e34e3f27e7678025586f49f31b8126dbb86cca558b26710eafefb34b0f130"} Oct 30 06:59:09.786841 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.786818 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:59:09.862363 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.862338 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-envoy\") pod \"5f2aec32-15b0-4a59-839b-a1e23f337319\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " Oct 30 06:59:09.862515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.862396 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2aec32-15b0-4a59-839b-a1e23f337319-kube-api-access-nc5pn\" (UniqueName: \"kubernetes.io/projected/5f2aec32-15b0-4a59-839b-a1e23f337319-kube-api-access-nc5pn\") pod \"5f2aec32-15b0-4a59-839b-a1e23f337319\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " Oct 30 06:59:09.862515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.862414 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-workload-certs\") pod \"5f2aec32-15b0-4a59-839b-a1e23f337319\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " Oct 30 06:59:09.862515 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.862477 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-credential-socket\") pod \"5f2aec32-15b0-4a59-839b-a1e23f337319\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " Oct 30 06:59:09.862670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.862517 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-podinfo\") pod \"5f2aec32-15b0-4a59-839b-a1e23f337319\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " Oct 30 06:59:09.862670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.862561 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-data\") pod \"5f2aec32-15b0-4a59-839b-a1e23f337319\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " Oct 30 06:59:09.862670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.862591 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2aec32-15b0-4a59-839b-a1e23f337319-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f2aec32-15b0-4a59-839b-a1e23f337319-istiod-ca-cert\") pod \"5f2aec32-15b0-4a59-839b-a1e23f337319\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " Oct 30 06:59:09.862871 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.862699 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-workload-socket\") pod \"5f2aec32-15b0-4a59-839b-a1e23f337319\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " Oct 30 06:59:09.862871 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.862758 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-token\" (UniqueName: \"kubernetes.io/projected/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-token\") pod \"5f2aec32-15b0-4a59-839b-a1e23f337319\" (UID: \"5f2aec32-15b0-4a59-839b-a1e23f337319\") " Oct 30 06:59:09.862871 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.862854 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-credential-socket" pod "5f2aec32-15b0-4a59-839b-a1e23f337319" (UID: "5f2aec32-15b0-4a59-839b-a1e23f337319"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:09.863457 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.863094 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-data" pod "5f2aec32-15b0-4a59-839b-a1e23f337319" (UID: "5f2aec32-15b0-4a59-839b-a1e23f337319"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:09.863457 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.863306 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-workload-socket" pod "5f2aec32-15b0-4a59-839b-a1e23f337319" (UID: "5f2aec32-15b0-4a59-839b-a1e23f337319"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:09.863457 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.863351 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2aec32-15b0-4a59-839b-a1e23f337319-istiod-ca-cert" pod "5f2aec32-15b0-4a59-839b-a1e23f337319" (UID: "5f2aec32-15b0-4a59-839b-a1e23f337319"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:09.863960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.863551 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.863960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.863576 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.863960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.863589 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f2aec32-15b0-4a59-839b-a1e23f337319-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.863960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.863596 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-workload-certs" pod "5f2aec32-15b0-4a59-839b-a1e23f337319" (UID: "5f2aec32-15b0-4a59-839b-a1e23f337319"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:09.863960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.863601 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.865148 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.865118 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-envoy" pod "5f2aec32-15b0-4a59-839b-a1e23f337319" (UID: "5f2aec32-15b0-4a59-839b-a1e23f337319"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:09.865148 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.865139 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2aec32-15b0-4a59-839b-a1e23f337319-kube-api-access-nc5pn" pod "5f2aec32-15b0-4a59-839b-a1e23f337319" (UID: "5f2aec32-15b0-4a59-839b-a1e23f337319"). InnerVolumeSpecName "kube-api-access-nc5pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:09.865446 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.865424 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-podinfo" pod "5f2aec32-15b0-4a59-839b-a1e23f337319" (UID: "5f2aec32-15b0-4a59-839b-a1e23f337319"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:59:09.865905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.865877 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-token" pod "5f2aec32-15b0-4a59-839b-a1e23f337319" (UID: "5f2aec32-15b0-4a59-839b-a1e23f337319"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:09.964228 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.964190 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.964367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.964236 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc5pn\" (UniqueName: \"kubernetes.io/projected/5f2aec32-15b0-4a59-839b-a1e23f337319-kube-api-access-nc5pn\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.964367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.964252 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f2aec32-15b0-4a59-839b-a1e23f337319-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.964367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.964264 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:09.964367 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:09.964276 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f2aec32-15b0-4a59-839b-a1e23f337319-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:10.661320 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:10.661289 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" Oct 30 06:59:10.661823 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:10.661319 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf" event={"ID":"5f2aec32-15b0-4a59-839b-a1e23f337319","Type":"ContainerDied","Data":"7dbabb1f7c76a74ec86efc5dfbe49a9563f5c335fd51da29e91526ff397734c2"} Oct 30 06:59:10.661823 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:10.661371 6999 scope.go:122] "RemoveContainer" containerID="e61e34e3f27e7678025586f49f31b8126dbb86cca558b26710eafefb34b0f130" Oct 30 06:59:10.715645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:10.715619 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf"] Oct 30 06:59:10.727812 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:10.727788 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-qtzdf"] Oct 30 06:59:10.758764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:10.758727 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:11.608138 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:11.608108 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2aec32-15b0-4a59-839b-a1e23f337319" path="/var/lib/kubelet/pods/5f2aec32-15b0-4a59-839b-a1e23f337319/volumes" Oct 30 06:59:11.862321 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:11.862235 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:12.007868 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:12.007837 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:12.497587 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:12.497553 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:12.749702 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:12.749616 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:12.885292 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:12.885260 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:13.208689 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:13.208649 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:14.419083 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:14.419047 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:14.805111 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:14.805032 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:14.952487 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:14.952458 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:16.349817 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:16.349787 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:18.124923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:18.124888 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:19.525427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.525387 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj"] Oct 30 06:59:19.525873 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.525626 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" podUID="70b7d49d-e508-4a13-bc59-06a17f7e17fe" containerName="maas-api" containerID="cri-o://fa79f8eb68f41b6e0041b1ce6c86758243be0daa91d076543f788a0c25750aad" gracePeriod=30 Oct 30 06:59:19.578767 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.578741 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c"] Oct 30 06:59:19.579023 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.578985 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" podUID="85d15937-bbc6-440d-ba22-f7989f1e774d" containerName="istio-proxy" containerID="cri-o://7b28bb76c6d0083666ebe47272cf92459120bcd7d193d1539fe6cad17563bcf3" gracePeriod=30 Oct 30 06:59:19.626684 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.626602 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:19.711913 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.711884 6999 generic.go:356] "Generic (PLEG): container finished" podID="70b7d49d-e508-4a13-bc59-06a17f7e17fe" containerID="fa79f8eb68f41b6e0041b1ce6c86758243be0daa91d076543f788a0c25750aad" exitCode=0 Oct 30 06:59:19.712133 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.712109 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" event={"ID":"70b7d49d-e508-4a13-bc59-06a17f7e17fe","Type":"ContainerDied","Data":"fa79f8eb68f41b6e0041b1ce6c86758243be0daa91d076543f788a0c25750aad"} Oct 30 06:59:19.789005 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.788945 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:59:19.871496 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.871462 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70b7d49d-e508-4a13-bc59-06a17f7e17fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b7d49d-e508-4a13-bc59-06a17f7e17fe-maas-api-tls\") pod \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " Oct 30 06:59:19.871658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.871575 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70b7d49d-e508-4a13-bc59-06a17f7e17fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b7d49d-e508-4a13-bc59-06a17f7e17fe-metrics-certs\") pod \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " Oct 30 06:59:19.871658 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.871611 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70b7d49d-e508-4a13-bc59-06a17f7e17fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b7d49d-e508-4a13-bc59-06a17f7e17fe-openshift-service-ca\") pod \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " Oct 30 06:59:19.871742 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.871692 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b7d49d-e508-4a13-bc59-06a17f7e17fe-kube-api-access-97nzp\" (UniqueName: \"kubernetes.io/projected/70b7d49d-e508-4a13-bc59-06a17f7e17fe-kube-api-access-97nzp\") pod \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\" (UID: \"70b7d49d-e508-4a13-bc59-06a17f7e17fe\") " Oct 30 06:59:19.872033 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.872008 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b7d49d-e508-4a13-bc59-06a17f7e17fe-openshift-service-ca" pod "70b7d49d-e508-4a13-bc59-06a17f7e17fe" (UID: "70b7d49d-e508-4a13-bc59-06a17f7e17fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:19.873774 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.873744 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b7d49d-e508-4a13-bc59-06a17f7e17fe-maas-api-tls" pod "70b7d49d-e508-4a13-bc59-06a17f7e17fe" (UID: "70b7d49d-e508-4a13-bc59-06a17f7e17fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:19.873876 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.873816 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b7d49d-e508-4a13-bc59-06a17f7e17fe-metrics-certs" pod "70b7d49d-e508-4a13-bc59-06a17f7e17fe" (UID: "70b7d49d-e508-4a13-bc59-06a17f7e17fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:19.874077 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.874050 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b7d49d-e508-4a13-bc59-06a17f7e17fe-kube-api-access-97nzp" pod "70b7d49d-e508-4a13-bc59-06a17f7e17fe" (UID: "70b7d49d-e508-4a13-bc59-06a17f7e17fe"). InnerVolumeSpecName "kube-api-access-97nzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:19.926886 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.926860 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd"] Oct 30 06:59:19.927274 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.927072 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" podUID="7fc83c12-1924-49f9-9725-901610b7ed41" containerName="maas-api" containerID="cri-o://1db50eb82fe58468f79f9ab17e291fd128a47648d8f2a8ce52d5f9e56f5b20ad" gracePeriod=30 Oct 30 06:59:19.973507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.973477 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70b7d49d-e508-4a13-bc59-06a17f7e17fe-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:19.973507 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.973505 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70b7d49d-e508-4a13-bc59-06a17f7e17fe-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:19.973696 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.973518 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70b7d49d-e508-4a13-bc59-06a17f7e17fe-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:19.973696 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:19.973530 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97nzp\" (UniqueName: \"kubernetes.io/projected/70b7d49d-e508-4a13-bc59-06a17f7e17fe-kube-api-access-97nzp\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:20.016505 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.016473 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:20.153094 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.153057 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:20.187873 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.187850 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:59:20.276182 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.276151 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fc83c12-1924-49f9-9725-901610b7ed41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc83c12-1924-49f9-9725-901610b7ed41-metrics-certs\") pod \"7fc83c12-1924-49f9-9725-901610b7ed41\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " Oct 30 06:59:20.276353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.276250 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc83c12-1924-49f9-9725-901610b7ed41-kube-api-access-r6mlt\" (UniqueName: \"kubernetes.io/projected/7fc83c12-1924-49f9-9725-901610b7ed41-kube-api-access-r6mlt\") pod \"7fc83c12-1924-49f9-9725-901610b7ed41\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " Oct 30 06:59:20.276353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.276326 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fc83c12-1924-49f9-9725-901610b7ed41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fc83c12-1924-49f9-9725-901610b7ed41-maas-api-tls\") pod \"7fc83c12-1924-49f9-9725-901610b7ed41\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " Oct 30 06:59:20.276442 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.276372 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc83c12-1924-49f9-9725-901610b7ed41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc83c12-1924-49f9-9725-901610b7ed41-openshift-service-ca\") pod \"7fc83c12-1924-49f9-9725-901610b7ed41\" (UID: \"7fc83c12-1924-49f9-9725-901610b7ed41\") " Oct 30 06:59:20.276802 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.276774 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc83c12-1924-49f9-9725-901610b7ed41-openshift-service-ca" pod "7fc83c12-1924-49f9-9725-901610b7ed41" (UID: "7fc83c12-1924-49f9-9725-901610b7ed41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:20.278473 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.278450 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fc83c12-1924-49f9-9725-901610b7ed41-maas-api-tls" pod "7fc83c12-1924-49f9-9725-901610b7ed41" (UID: "7fc83c12-1924-49f9-9725-901610b7ed41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:20.278563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.278478 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fc83c12-1924-49f9-9725-901610b7ed41-metrics-certs" pod "7fc83c12-1924-49f9-9725-901610b7ed41" (UID: "7fc83c12-1924-49f9-9725-901610b7ed41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:20.278563 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.278509 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc83c12-1924-49f9-9725-901610b7ed41-kube-api-access-r6mlt" pod "7fc83c12-1924-49f9-9725-901610b7ed41" (UID: "7fc83c12-1924-49f9-9725-901610b7ed41"). InnerVolumeSpecName "kube-api-access-r6mlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:20.298550 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.298493 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:20.377763 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.377731 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fc83c12-1924-49f9-9725-901610b7ed41-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:20.377763 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.377764 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6mlt\" (UniqueName: \"kubernetes.io/projected/7fc83c12-1924-49f9-9725-901610b7ed41-kube-api-access-r6mlt\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:20.378057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.377776 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fc83c12-1924-49f9-9725-901610b7ed41-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:20.378057 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.377786 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc83c12-1924-49f9-9725-901610b7ed41-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:20.672470 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.672442 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:20.727369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.718443 6999 generic.go:356] "Generic (PLEG): container finished" podID="7fc83c12-1924-49f9-9725-901610b7ed41" containerID="1db50eb82fe58468f79f9ab17e291fd128a47648d8f2a8ce52d5f9e56f5b20ad" exitCode=0 Oct 30 06:59:20.727369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.718583 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" Oct 30 06:59:20.727369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.719108 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" event={"ID":"7fc83c12-1924-49f9-9725-901610b7ed41","Type":"ContainerDied","Data":"1db50eb82fe58468f79f9ab17e291fd128a47648d8f2a8ce52d5f9e56f5b20ad"} Oct 30 06:59:20.727369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.719178 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd" event={"ID":"7fc83c12-1924-49f9-9725-901610b7ed41","Type":"ContainerDied","Data":"acdc4b4184119e2add454d69b45aa223275ff8ade68c2a7d4e020bf78f30218f"} Oct 30 06:59:20.727369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.719209 6999 scope.go:122] "RemoveContainer" containerID="1db50eb82fe58468f79f9ab17e291fd128a47648d8f2a8ce52d5f9e56f5b20ad" Oct 30 06:59:20.727369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.720685 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" event={"ID":"70b7d49d-e508-4a13-bc59-06a17f7e17fe","Type":"ContainerDied","Data":"b16f14252ae184b9c625ae43b360fbe98c8fe35b1ee1beba6315bfa8e51dd4bd"} Oct 30 06:59:20.727369 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.720775 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj" Oct 30 06:59:20.728897 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.728854 6999 scope.go:122] "RemoveContainer" containerID="1db50eb82fe58468f79f9ab17e291fd128a47648d8f2a8ce52d5f9e56f5b20ad" Oct 30 06:59:20.729268 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:59:20.729136 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db50eb82fe58468f79f9ab17e291fd128a47648d8f2a8ce52d5f9e56f5b20ad\": container with ID starting with 1db50eb82fe58468f79f9ab17e291fd128a47648d8f2a8ce52d5f9e56f5b20ad not found: ID does not exist" containerID="1db50eb82fe58468f79f9ab17e291fd128a47648d8f2a8ce52d5f9e56f5b20ad" Oct 30 06:59:20.729268 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.729170 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db50eb82fe58468f79f9ab17e291fd128a47648d8f2a8ce52d5f9e56f5b20ad"} err="failed to get container status \"1db50eb82fe58468f79f9ab17e291fd128a47648d8f2a8ce52d5f9e56f5b20ad\": rpc error: code = NotFound desc = could not find container \"1db50eb82fe58468f79f9ab17e291fd128a47648d8f2a8ce52d5f9e56f5b20ad\": container with ID starting with 1db50eb82fe58468f79f9ab17e291fd128a47648d8f2a8ce52d5f9e56f5b20ad not found: ID does not exist" Oct 30 06:59:20.729268 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.729194 6999 scope.go:122] "RemoveContainer" containerID="fa79f8eb68f41b6e0041b1ce6c86758243be0daa91d076543f788a0c25750aad" Oct 30 06:59:20.772448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.772425 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd"] Oct 30 06:59:20.783427 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.778489 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-69f3fa-59bb896c8c-c44sd"] Oct 30 06:59:20.809556 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.809524 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:20.811273 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.811247 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj"] Oct 30 06:59:20.813919 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.813902 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2d8df2-7c8766f46b-wqnnj"] Oct 30 06:59:20.956495 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:20.956396 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:21.387589 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:21.387554 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:21.608640 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:21.608613 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70b7d49d-e508-4a13-bc59-06a17f7e17fe" path="/var/lib/kubelet/pods/70b7d49d-e508-4a13-bc59-06a17f7e17fe/volumes" Oct 30 06:59:21.609015 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:21.609002 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc83c12-1924-49f9-9725-901610b7ed41" path="/var/lib/kubelet/pods/7fc83c12-1924-49f9-9725-901610b7ed41/volumes" Oct 30 06:59:22.671548 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:22.671512 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:24.205560 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.205510 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll"] Oct 30 06:59:24.348739 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.348704 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:24.491450 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.491333 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:24.747706 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.747626 6999 generic.go:356] "Generic (PLEG): container finished" podID="85d15937-bbc6-440d-ba22-f7989f1e774d" containerID="7b28bb76c6d0083666ebe47272cf92459120bcd7d193d1539fe6cad17563bcf3" exitCode=0 Oct 30 06:59:24.747873 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.747738 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" event={"ID":"85d15937-bbc6-440d-ba22-f7989f1e774d","Type":"ContainerDied","Data":"7b28bb76c6d0083666ebe47272cf92459120bcd7d193d1539fe6cad17563bcf3"} Oct 30 06:59:24.863670 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.863643 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:59:24.935275 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.935240 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-envoy\") pod \"85d15937-bbc6-440d-ba22-f7989f1e774d\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " Oct 30 06:59:24.935460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.935286 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d15937-bbc6-440d-ba22-f7989f1e774d-kube-api-access-pm9qj\" (UniqueName: \"kubernetes.io/projected/85d15937-bbc6-440d-ba22-f7989f1e774d-kube-api-access-pm9qj\") pod \"85d15937-bbc6-440d-ba22-f7989f1e774d\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " Oct 30 06:59:24.935460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.935336 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-credential-socket\") pod \"85d15937-bbc6-440d-ba22-f7989f1e774d\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " Oct 30 06:59:24.935460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.935375 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-token\" (UniqueName: \"kubernetes.io/projected/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-token\") pod \"85d15937-bbc6-440d-ba22-f7989f1e774d\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " Oct 30 06:59:24.935460 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.935413 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-workload-socket\") pod \"85d15937-bbc6-440d-ba22-f7989f1e774d\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " Oct 30 06:59:24.935685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.935462 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-data\") pod \"85d15937-bbc6-440d-ba22-f7989f1e774d\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " Oct 30 06:59:24.935685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.935522 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-podinfo\") pod \"85d15937-bbc6-440d-ba22-f7989f1e774d\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " Oct 30 06:59:24.935685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.935547 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d15937-bbc6-440d-ba22-f7989f1e774d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d15937-bbc6-440d-ba22-f7989f1e774d-istiod-ca-cert\") pod \"85d15937-bbc6-440d-ba22-f7989f1e774d\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " Oct 30 06:59:24.935685 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.935582 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-workload-certs\") pod \"85d15937-bbc6-440d-ba22-f7989f1e774d\" (UID: \"85d15937-bbc6-440d-ba22-f7989f1e774d\") " Oct 30 06:59:24.935902 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.935687 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-credential-socket" pod "85d15937-bbc6-440d-ba22-f7989f1e774d" (UID: "85d15937-bbc6-440d-ba22-f7989f1e774d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:24.935971 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.935912 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-workload-certs" pod "85d15937-bbc6-440d-ba22-f7989f1e774d" (UID: "85d15937-bbc6-440d-ba22-f7989f1e774d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:24.936488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.936109 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:24.936488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.936134 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:24.936488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.936146 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-data" pod "85d15937-bbc6-440d-ba22-f7989f1e774d" (UID: "85d15937-bbc6-440d-ba22-f7989f1e774d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:24.936488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.936252 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-workload-socket" pod "85d15937-bbc6-440d-ba22-f7989f1e774d" (UID: "85d15937-bbc6-440d-ba22-f7989f1e774d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:24.936488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.936438 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d15937-bbc6-440d-ba22-f7989f1e774d-istiod-ca-cert" pod "85d15937-bbc6-440d-ba22-f7989f1e774d" (UID: "85d15937-bbc6-440d-ba22-f7989f1e774d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:24.938231 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.938185 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-token" pod "85d15937-bbc6-440d-ba22-f7989f1e774d" (UID: "85d15937-bbc6-440d-ba22-f7989f1e774d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:24.938830 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.938802 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d15937-bbc6-440d-ba22-f7989f1e774d-kube-api-access-pm9qj" pod "85d15937-bbc6-440d-ba22-f7989f1e774d" (UID: "85d15937-bbc6-440d-ba22-f7989f1e774d"). InnerVolumeSpecName "kube-api-access-pm9qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:24.939716 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.939690 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-envoy" pod "85d15937-bbc6-440d-ba22-f7989f1e774d" (UID: "85d15937-bbc6-440d-ba22-f7989f1e774d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:59:24.939799 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:24.939775 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-podinfo" pod "85d15937-bbc6-440d-ba22-f7989f1e774d" (UID: "85d15937-bbc6-440d-ba22-f7989f1e774d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:59:25.037594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:25.037517 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:25.037594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:25.037544 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm9qj\" (UniqueName: \"kubernetes.io/projected/85d15937-bbc6-440d-ba22-f7989f1e774d-kube-api-access-pm9qj\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:25.037594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:25.037554 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:25.037594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:25.037565 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:25.037594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:25.037572 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:25.037594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:25.037580 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85d15937-bbc6-440d-ba22-f7989f1e774d-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:25.037594 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:25.037587 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85d15937-bbc6-440d-ba22-f7989f1e774d-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:25.754413 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:25.754370 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" Oct 30 06:59:25.754868 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:25.754407 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c" event={"ID":"85d15937-bbc6-440d-ba22-f7989f1e774d","Type":"ContainerDied","Data":"7d4b44a9f2a1673c27af99fc11c85a336f9d77fcad7f1dd079a06581dd1229eb"} Oct 30 06:59:25.754868 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:25.754467 6999 scope.go:122] "RemoveContainer" containerID="7b28bb76c6d0083666ebe47272cf92459120bcd7d193d1539fe6cad17563bcf3" Oct 30 06:59:25.818263 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:25.818233 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c"] Oct 30 06:59:25.821109 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:25.821085 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-b9r4c"] Oct 30 06:59:26.118048 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:26.118022 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:27.526084 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:27.526037 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:27.609575 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:27.609533 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d15937-bbc6-440d-ba22-f7989f1e774d" path="/var/lib/kubelet/pods/85d15937-bbc6-440d-ba22-f7989f1e774d/volumes" Oct 30 06:59:27.669471 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:27.669435 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:27.920078 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:27.920048 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:28.065383 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:28.065352 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:28.491375 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:28.491340 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:28.635298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:28.635260 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:28.866243 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:28.866198 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:29.206498 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:29.206414 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" podUID="7bc29aa6-9b21-4581-94f3-3e5302cc4b6b" containerName="payload-pre-processing" containerID="cri-o://2ad8b9d3c136fc93ec2f6a9625f88246b841ae5e5e51a134c4f88cbb3aef1545" gracePeriod=25 Oct 30 06:59:29.909436 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:29.909403 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:31.729569 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:31.729528 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:33.533298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:33.533251 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:34.142834 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:59:34.142797 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 06:59:34.632298 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:34.632266 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" podUID="7bc29aa6-9b21-4581-94f3-3e5302cc4b6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:35.331869 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:35.331816 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:37.039731 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:37.039698 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:38.411388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.411356 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b"] Oct 30 06:59:38.411912 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.411591 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" podUID="67c137ac-abb6-412a-9dac-7639c95abaa0" containerName="maas-api" containerID="cri-o://6f35d4a0ac5ab820010204fc9e4e4485893940e27bf6a633e64fa1c391d68f6e" gracePeriod=30 Oct 30 06:59:38.510826 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.510790 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:38.704678 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.704653 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:59:38.788683 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.788645 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-maas-api-tls\") pod \"67c137ac-abb6-412a-9dac-7639c95abaa0\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " Oct 30 06:59:38.788923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.788728 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c137ac-abb6-412a-9dac-7639c95abaa0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67c137ac-abb6-412a-9dac-7639c95abaa0-openshift-service-ca\") pod \"67c137ac-abb6-412a-9dac-7639c95abaa0\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " Oct 30 06:59:38.788923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.788785 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-metrics-certs\") pod \"67c137ac-abb6-412a-9dac-7639c95abaa0\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " Oct 30 06:59:38.788923 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.788884 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c137ac-abb6-412a-9dac-7639c95abaa0-kube-api-access-76bkd\" (UniqueName: \"kubernetes.io/projected/67c137ac-abb6-412a-9dac-7639c95abaa0-kube-api-access-76bkd\") pod \"67c137ac-abb6-412a-9dac-7639c95abaa0\" (UID: \"67c137ac-abb6-412a-9dac-7639c95abaa0\") " Oct 30 06:59:38.789202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.789165 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c137ac-abb6-412a-9dac-7639c95abaa0-openshift-service-ca" pod "67c137ac-abb6-412a-9dac-7639c95abaa0" (UID: "67c137ac-abb6-412a-9dac-7639c95abaa0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:38.789377 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.789334 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67c137ac-abb6-412a-9dac-7639c95abaa0-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:38.791308 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.791280 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-maas-api-tls" pod "67c137ac-abb6-412a-9dac-7639c95abaa0" (UID: "67c137ac-abb6-412a-9dac-7639c95abaa0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:38.791532 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.791502 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-metrics-certs" pod "67c137ac-abb6-412a-9dac-7639c95abaa0" (UID: "67c137ac-abb6-412a-9dac-7639c95abaa0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:38.791645 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.791575 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c137ac-abb6-412a-9dac-7639c95abaa0-kube-api-access-76bkd" pod "67c137ac-abb6-412a-9dac-7639c95abaa0" (UID: "67c137ac-abb6-412a-9dac-7639c95abaa0"). InnerVolumeSpecName "kube-api-access-76bkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:38.828799 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.828765 6999 generic.go:356] "Generic (PLEG): container finished" podID="67c137ac-abb6-412a-9dac-7639c95abaa0" containerID="6f35d4a0ac5ab820010204fc9e4e4485893940e27bf6a633e64fa1c391d68f6e" exitCode=0 Oct 30 06:59:38.828911 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.828835 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" Oct 30 06:59:38.828911 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.828852 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" event={"ID":"67c137ac-abb6-412a-9dac-7639c95abaa0","Type":"ContainerDied","Data":"6f35d4a0ac5ab820010204fc9e4e4485893940e27bf6a633e64fa1c391d68f6e"} Oct 30 06:59:38.829002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.828926 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b" event={"ID":"67c137ac-abb6-412a-9dac-7639c95abaa0","Type":"ContainerDied","Data":"08fcbbf14aa666d385685491c1e0830c74205e3f1ddae18a4bd2acf54cb853ef"} Oct 30 06:59:38.829002 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.828949 6999 scope.go:122] "RemoveContainer" containerID="6f35d4a0ac5ab820010204fc9e4e4485893940e27bf6a633e64fa1c391d68f6e" Oct 30 06:59:38.839264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.839202 6999 scope.go:122] "RemoveContainer" containerID="6f35d4a0ac5ab820010204fc9e4e4485893940e27bf6a633e64fa1c391d68f6e" Oct 30 06:59:38.839556 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:59:38.839537 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f35d4a0ac5ab820010204fc9e4e4485893940e27bf6a633e64fa1c391d68f6e\": container with ID starting with 6f35d4a0ac5ab820010204fc9e4e4485893940e27bf6a633e64fa1c391d68f6e not found: ID does not exist" containerID="6f35d4a0ac5ab820010204fc9e4e4485893940e27bf6a633e64fa1c391d68f6e" Oct 30 06:59:38.839631 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.839567 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f35d4a0ac5ab820010204fc9e4e4485893940e27bf6a633e64fa1c391d68f6e"} err="failed to get container status \"6f35d4a0ac5ab820010204fc9e4e4485893940e27bf6a633e64fa1c391d68f6e\": rpc error: code = NotFound desc = could not find container \"6f35d4a0ac5ab820010204fc9e4e4485893940e27bf6a633e64fa1c391d68f6e\": container with ID starting with 6f35d4a0ac5ab820010204fc9e4e4485893940e27bf6a633e64fa1c391d68f6e not found: ID does not exist" Oct 30 06:59:38.880765 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.880734 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b"] Oct 30 06:59:38.889321 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.889286 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-6e2548-799c9df587-k7h4b"] Oct 30 06:59:38.890623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.890596 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76bkd\" (UniqueName: \"kubernetes.io/projected/67c137ac-abb6-412a-9dac-7639c95abaa0-kube-api-access-76bkd\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:38.890623 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.890623 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:38.890791 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:38.890639 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67c137ac-abb6-412a-9dac-7639c95abaa0-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:39.034634 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:39.034549 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:39.185104 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:39.185061 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:39.608776 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:39.608746 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c137ac-abb6-412a-9dac-7639c95abaa0" path="/var/lib/kubelet/pods/67c137ac-abb6-412a-9dac-7639c95abaa0/volumes" Oct 30 06:59:39.725396 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:39.725361 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:39.866947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:39.866871 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:40.945853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:40.945823 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:42.333015 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.332981 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t"] Oct 30 06:59:42.333528 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.333277 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" podUID="3e6752f6-ee5f-4800-8fd8-b9df6aa69274" containerName="maas-api" containerID="cri-o://391e065825910030537fb1981c4a0cfab155a3f5246fb66edb1bb3f4ccfa721f" gracePeriod=30 Oct 30 06:59:42.432353 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.432316 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:42.603035 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.603008 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:59:42.732001 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.731965 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-openshift-service-ca\") pod \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " Oct 30 06:59:42.732174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.732102 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-maas-api-tls\") pod \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " Oct 30 06:59:42.732174 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.732163 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-kube-api-access-4rjpf\" (UniqueName: \"kubernetes.io/projected/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-kube-api-access-4rjpf\") pod \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " Oct 30 06:59:42.732324 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.732191 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-metrics-certs\") pod \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\" (UID: \"3e6752f6-ee5f-4800-8fd8-b9df6aa69274\") " Oct 30 06:59:42.733688 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.732453 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-openshift-service-ca" pod "3e6752f6-ee5f-4800-8fd8-b9df6aa69274" (UID: "3e6752f6-ee5f-4800-8fd8-b9df6aa69274"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:42.733688 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.733399 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:42.734737 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.734709 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-maas-api-tls" pod "3e6752f6-ee5f-4800-8fd8-b9df6aa69274" (UID: "3e6752f6-ee5f-4800-8fd8-b9df6aa69274"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:42.734834 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.734772 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-kube-api-access-4rjpf" pod "3e6752f6-ee5f-4800-8fd8-b9df6aa69274" (UID: "3e6752f6-ee5f-4800-8fd8-b9df6aa69274"). InnerVolumeSpecName "kube-api-access-4rjpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:42.735264 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.735238 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-metrics-certs" pod "3e6752f6-ee5f-4800-8fd8-b9df6aa69274" (UID: "3e6752f6-ee5f-4800-8fd8-b9df6aa69274"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:42.834526 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.834452 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:42.834526 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.834486 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rjpf\" (UniqueName: \"kubernetes.io/projected/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-kube-api-access-4rjpf\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:42.834526 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.834498 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e6752f6-ee5f-4800-8fd8-b9df6aa69274-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:42.853056 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.853026 6999 generic.go:356] "Generic (PLEG): container finished" podID="3e6752f6-ee5f-4800-8fd8-b9df6aa69274" containerID="391e065825910030537fb1981c4a0cfab155a3f5246fb66edb1bb3f4ccfa721f" exitCode=0 Oct 30 06:59:42.853202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.853075 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" Oct 30 06:59:42.853202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.853090 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" event={"ID":"3e6752f6-ee5f-4800-8fd8-b9df6aa69274","Type":"ContainerDied","Data":"391e065825910030537fb1981c4a0cfab155a3f5246fb66edb1bb3f4ccfa721f"} Oct 30 06:59:42.853202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.853133 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t" event={"ID":"3e6752f6-ee5f-4800-8fd8-b9df6aa69274","Type":"ContainerDied","Data":"dd8663bba8887d332c100e8abf772eff3d339c3c5e497c023dfb82f63b1d8d97"} Oct 30 06:59:42.853202 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.853161 6999 scope.go:122] "RemoveContainer" containerID="391e065825910030537fb1981c4a0cfab155a3f5246fb66edb1bb3f4ccfa721f" Oct 30 06:59:42.863781 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.863765 6999 scope.go:122] "RemoveContainer" containerID="391e065825910030537fb1981c4a0cfab155a3f5246fb66edb1bb3f4ccfa721f" Oct 30 06:59:42.864028 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:59:42.864009 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391e065825910030537fb1981c4a0cfab155a3f5246fb66edb1bb3f4ccfa721f\": container with ID starting with 391e065825910030537fb1981c4a0cfab155a3f5246fb66edb1bb3f4ccfa721f not found: ID does not exist" containerID="391e065825910030537fb1981c4a0cfab155a3f5246fb66edb1bb3f4ccfa721f" Oct 30 06:59:42.864093 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.864038 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391e065825910030537fb1981c4a0cfab155a3f5246fb66edb1bb3f4ccfa721f"} err="failed to get container status \"391e065825910030537fb1981c4a0cfab155a3f5246fb66edb1bb3f4ccfa721f\": rpc error: code = NotFound desc = could not find container \"391e065825910030537fb1981c4a0cfab155a3f5246fb66edb1bb3f4ccfa721f\": container with ID starting with 391e065825910030537fb1981c4a0cfab155a3f5246fb66edb1bb3f4ccfa721f not found: ID does not exist" Oct 30 06:59:42.904351 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.904324 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t"] Oct 30 06:59:42.907960 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:42.907940 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-645f2b-6f79c77dcf-bjt4t"] Oct 30 06:59:43.053901 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:43.053839 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:43.221401 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:43.221373 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:43.608125 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:43.608051 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6752f6-ee5f-4800-8fd8-b9df6aa69274" path="/var/lib/kubelet/pods/3e6752f6-ee5f-4800-8fd8-b9df6aa69274/volumes" Oct 30 06:59:43.712378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:43.712347 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:44.632356 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:44.632323 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" podUID="7bc29aa6-9b21-4581-94f3-3e5302cc4b6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:59:45.134399 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:45.134344 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:46.451153 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:46.451097 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:46.596887 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:46.596855 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:46.815448 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:46.815365 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:46.967617 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:46.967590 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:48.741621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:48.741588 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:49.460099 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.460078 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" Oct 30 06:59:49.498044 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.497983 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc29aa6-9b21-4581-94f3-3e5302cc4b6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc29aa6-9b21-4581-94f3-3e5302cc4b6b-plugins-config-volume\") pod \"7bc29aa6-9b21-4581-94f3-3e5302cc4b6b\" (UID: \"7bc29aa6-9b21-4581-94f3-3e5302cc4b6b\") " Oct 30 06:59:49.498183 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.498077 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc29aa6-9b21-4581-94f3-3e5302cc4b6b-kube-api-access-mpvg4\" (UniqueName: \"kubernetes.io/projected/7bc29aa6-9b21-4581-94f3-3e5302cc4b6b-kube-api-access-mpvg4\") pod \"7bc29aa6-9b21-4581-94f3-3e5302cc4b6b\" (UID: \"7bc29aa6-9b21-4581-94f3-3e5302cc4b6b\") " Oct 30 06:59:49.498425 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.498404 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc29aa6-9b21-4581-94f3-3e5302cc4b6b-plugins-config-volume" pod "7bc29aa6-9b21-4581-94f3-3e5302cc4b6b" (UID: "7bc29aa6-9b21-4581-94f3-3e5302cc4b6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:49.498511 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.498498 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc29aa6-9b21-4581-94f3-3e5302cc4b6b-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:49.500456 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.500435 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc29aa6-9b21-4581-94f3-3e5302cc4b6b-kube-api-access-mpvg4" pod "7bc29aa6-9b21-4581-94f3-3e5302cc4b6b" (UID: "7bc29aa6-9b21-4581-94f3-3e5302cc4b6b"). InnerVolumeSpecName "kube-api-access-mpvg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:49.599276 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.599247 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpvg4\" (UniqueName: \"kubernetes.io/projected/7bc29aa6-9b21-4581-94f3-3e5302cc4b6b-kube-api-access-mpvg4\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:49.897378 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.897344 6999 generic.go:356] "Generic (PLEG): container finished" podID="7bc29aa6-9b21-4581-94f3-3e5302cc4b6b" containerID="2ad8b9d3c136fc93ec2f6a9625f88246b841ae5e5e51a134c4f88cbb3aef1545" exitCode=0 Oct 30 06:59:49.897905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.897382 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" event={"ID":"7bc29aa6-9b21-4581-94f3-3e5302cc4b6b","Type":"ContainerDied","Data":"2ad8b9d3c136fc93ec2f6a9625f88246b841ae5e5e51a134c4f88cbb3aef1545"} Oct 30 06:59:49.897905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.897388 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" Oct 30 06:59:49.897905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.897433 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll" event={"ID":"7bc29aa6-9b21-4581-94f3-3e5302cc4b6b","Type":"ContainerDied","Data":"00d61b120212d77bc6ad25a0eb0cab97e54cd53cd2adbedd3ae6ddd1a85cd7a9"} Oct 30 06:59:49.897905 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.897464 6999 scope.go:122] "RemoveContainer" containerID="2ad8b9d3c136fc93ec2f6a9625f88246b841ae5e5e51a134c4f88cbb3aef1545" Oct 30 06:59:49.907047 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.907028 6999 scope.go:122] "RemoveContainer" containerID="2ad8b9d3c136fc93ec2f6a9625f88246b841ae5e5e51a134c4f88cbb3aef1545" Oct 30 06:59:49.907400 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:59:49.907383 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad8b9d3c136fc93ec2f6a9625f88246b841ae5e5e51a134c4f88cbb3aef1545\": container with ID starting with 2ad8b9d3c136fc93ec2f6a9625f88246b841ae5e5e51a134c4f88cbb3aef1545 not found: ID does not exist" containerID="2ad8b9d3c136fc93ec2f6a9625f88246b841ae5e5e51a134c4f88cbb3aef1545" Oct 30 06:59:49.907509 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.907407 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad8b9d3c136fc93ec2f6a9625f88246b841ae5e5e51a134c4f88cbb3aef1545"} err="failed to get container status \"2ad8b9d3c136fc93ec2f6a9625f88246b841ae5e5e51a134c4f88cbb3aef1545\": rpc error: code = NotFound desc = could not find container \"2ad8b9d3c136fc93ec2f6a9625f88246b841ae5e5e51a134c4f88cbb3aef1545\": container with ID starting with 2ad8b9d3c136fc93ec2f6a9625f88246b841ae5e5e51a134c4f88cbb3aef1545 not found: ID does not exist" Oct 30 06:59:49.947853 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.947833 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll"] Oct 30 06:59:49.958947 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:49.958925 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-6e2548-659965cc47-d7bll"] Oct 30 06:59:50.548938 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:50.548910 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:50.772886 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:50.772851 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:50.984143 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:50.984092 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:51.607310 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:51.607278 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc29aa6-9b21-4581-94f3-3e5302cc4b6b" path="/var/lib/kubelet/pods/7bc29aa6-9b21-4581-94f3-3e5302cc4b6b/volumes" Oct 30 06:59:52.448308 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:52.448272 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:53.960388 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:53.960358 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g"] Oct 30 06:59:53.960900 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:53.960571 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" podUID="dceee649-894a-4330-a9df-0097ca9e352e" containerName="maas-api" containerID="cri-o://d9b87b5c22c75cb1610b292e84c590d4b9fcbff701acce8a765c007ba69b3662" gracePeriod=30 Oct 30 06:59:54.062526 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.062492 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:54.243135 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.243102 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:59:54.347839 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.347812 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dceee649-894a-4330-a9df-0097ca9e352e-kube-api-access-c544z\" (UniqueName: \"kubernetes.io/projected/dceee649-894a-4330-a9df-0097ca9e352e-kube-api-access-c544z\") pod \"dceee649-894a-4330-a9df-0097ca9e352e\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " Oct 30 06:59:54.348032 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.347858 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dceee649-894a-4330-a9df-0097ca9e352e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dceee649-894a-4330-a9df-0097ca9e352e-openshift-service-ca\") pod \"dceee649-894a-4330-a9df-0097ca9e352e\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " Oct 30 06:59:54.348032 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.347903 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-metrics-certs\") pod \"dceee649-894a-4330-a9df-0097ca9e352e\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " Oct 30 06:59:54.348032 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.347952 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-maas-api-tls\") pod \"dceee649-894a-4330-a9df-0097ca9e352e\" (UID: \"dceee649-894a-4330-a9df-0097ca9e352e\") " Oct 30 06:59:54.348355 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.348329 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dceee649-894a-4330-a9df-0097ca9e352e-openshift-service-ca" pod "dceee649-894a-4330-a9df-0097ca9e352e" (UID: "dceee649-894a-4330-a9df-0097ca9e352e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:54.350134 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.350103 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-maas-api-tls" pod "dceee649-894a-4330-a9df-0097ca9e352e" (UID: "dceee649-894a-4330-a9df-0097ca9e352e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:54.350275 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.350170 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-metrics-certs" pod "dceee649-894a-4330-a9df-0097ca9e352e" (UID: "dceee649-894a-4330-a9df-0097ca9e352e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:54.350275 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.350244 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dceee649-894a-4330-a9df-0097ca9e352e-kube-api-access-c544z" pod "dceee649-894a-4330-a9df-0097ca9e352e" (UID: "dceee649-894a-4330-a9df-0097ca9e352e"). InnerVolumeSpecName "kube-api-access-c544z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:54.449621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.449579 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c544z\" (UniqueName: \"kubernetes.io/projected/dceee649-894a-4330-a9df-0097ca9e352e-kube-api-access-c544z\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:54.449621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.449614 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dceee649-894a-4330-a9df-0097ca9e352e-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:54.449621 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.449628 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:54.449868 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.449642 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dceee649-894a-4330-a9df-0097ca9e352e-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:54.611342 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.611310 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:54.760602 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.760576 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:54.928662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.928579 6999 generic.go:356] "Generic (PLEG): container finished" podID="dceee649-894a-4330-a9df-0097ca9e352e" containerID="d9b87b5c22c75cb1610b292e84c590d4b9fcbff701acce8a765c007ba69b3662" exitCode=0 Oct 30 06:59:54.928662 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.928624 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" Oct 30 06:59:54.928870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.928663 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" event={"ID":"dceee649-894a-4330-a9df-0097ca9e352e","Type":"ContainerDied","Data":"d9b87b5c22c75cb1610b292e84c590d4b9fcbff701acce8a765c007ba69b3662"} Oct 30 06:59:54.928870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.928716 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g" event={"ID":"dceee649-894a-4330-a9df-0097ca9e352e","Type":"ContainerDied","Data":"4f376b7f34ab57774afc6ca7f95ba37314a53f38d90acdf9637c031fea8505da"} Oct 30 06:59:54.928870 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.928745 6999 scope.go:122] "RemoveContainer" containerID="d9b87b5c22c75cb1610b292e84c590d4b9fcbff701acce8a765c007ba69b3662" Oct 30 06:59:54.939189 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.939170 6999 scope.go:122] "RemoveContainer" containerID="d9b87b5c22c75cb1610b292e84c590d4b9fcbff701acce8a765c007ba69b3662" Oct 30 06:59:54.939466 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:59:54.939444 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b87b5c22c75cb1610b292e84c590d4b9fcbff701acce8a765c007ba69b3662\": container with ID starting with d9b87b5c22c75cb1610b292e84c590d4b9fcbff701acce8a765c007ba69b3662 not found: ID does not exist" containerID="d9b87b5c22c75cb1610b292e84c590d4b9fcbff701acce8a765c007ba69b3662" Oct 30 06:59:54.939550 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.939479 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b87b5c22c75cb1610b292e84c590d4b9fcbff701acce8a765c007ba69b3662"} err="failed to get container status \"d9b87b5c22c75cb1610b292e84c590d4b9fcbff701acce8a765c007ba69b3662\": rpc error: code = NotFound desc = could not find container \"d9b87b5c22c75cb1610b292e84c590d4b9fcbff701acce8a765c007ba69b3662\": container with ID starting with d9b87b5c22c75cb1610b292e84c590d4b9fcbff701acce8a765c007ba69b3662 not found: ID does not exist" Oct 30 06:59:54.987801 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.987769 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g"] Oct 30 06:59:54.999582 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:54.999536 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-adcdcb-5b99699bdc-4ms4g"] Oct 30 06:59:55.304408 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:55.304329 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:55.607409 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:55.607338 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dceee649-894a-4330-a9df-0097ca9e352e" path="/var/lib/kubelet/pods/dceee649-894a-4330-a9df-0097ca9e352e/volumes" Oct 30 06:59:56.523453 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:56.523425 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:57.849454 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:57.849424 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7"] Oct 30 06:59:57.909303 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:57.909264 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85"] Oct 30 06:59:57.909488 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:57.909472 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" podUID="701a7be4-531f-434e-9e1e-a85db672c13f" containerName="maas-api" containerID="cri-o://806a2456ccab2a55ffa9b335b6c11aa96f351de2e665a8a9023502ac4debe58f" gracePeriod=30 Oct 30 06:59:58.002511 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.002474 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:58.139676 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.139609 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:58.178283 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.178258 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:59:58.289532 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.289500 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/701a7be4-531f-434e-9e1e-a85db672c13f-kube-api-access-7kznk\" (UniqueName: \"kubernetes.io/projected/701a7be4-531f-434e-9e1e-a85db672c13f-kube-api-access-7kznk\") pod \"701a7be4-531f-434e-9e1e-a85db672c13f\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " Oct 30 06:59:58.289706 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.289552 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/701a7be4-531f-434e-9e1e-a85db672c13f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/701a7be4-531f-434e-9e1e-a85db672c13f-maas-api-tls\") pod \"701a7be4-531f-434e-9e1e-a85db672c13f\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " Oct 30 06:59:58.289706 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.289666 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/701a7be4-531f-434e-9e1e-a85db672c13f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/701a7be4-531f-434e-9e1e-a85db672c13f-metrics-certs\") pod \"701a7be4-531f-434e-9e1e-a85db672c13f\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " Oct 30 06:59:58.289706 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.289684 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/701a7be4-531f-434e-9e1e-a85db672c13f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/701a7be4-531f-434e-9e1e-a85db672c13f-openshift-service-ca\") pod \"701a7be4-531f-434e-9e1e-a85db672c13f\" (UID: \"701a7be4-531f-434e-9e1e-a85db672c13f\") " Oct 30 06:59:58.290096 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.290069 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701a7be4-531f-434e-9e1e-a85db672c13f-openshift-service-ca" pod "701a7be4-531f-434e-9e1e-a85db672c13f" (UID: "701a7be4-531f-434e-9e1e-a85db672c13f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:59:58.291907 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.291878 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701a7be4-531f-434e-9e1e-a85db672c13f-kube-api-access-7kznk" pod "701a7be4-531f-434e-9e1e-a85db672c13f" (UID: "701a7be4-531f-434e-9e1e-a85db672c13f"). InnerVolumeSpecName "kube-api-access-7kznk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:59:58.292045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.291915 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/701a7be4-531f-434e-9e1e-a85db672c13f-maas-api-tls" pod "701a7be4-531f-434e-9e1e-a85db672c13f" (UID: "701a7be4-531f-434e-9e1e-a85db672c13f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:58.292045 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.291929 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/701a7be4-531f-434e-9e1e-a85db672c13f-metrics-certs" pod "701a7be4-531f-434e-9e1e-a85db672c13f" (UID: "701a7be4-531f-434e-9e1e-a85db672c13f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:59:58.390772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.390691 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kznk\" (UniqueName: \"kubernetes.io/projected/701a7be4-531f-434e-9e1e-a85db672c13f-kube-api-access-7kznk\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:58.390772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.390718 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/701a7be4-531f-434e-9e1e-a85db672c13f-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:58.390772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.390727 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/701a7be4-531f-434e-9e1e-a85db672c13f-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:58.390772 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.390735 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/701a7be4-531f-434e-9e1e-a85db672c13f-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 06:59:58.713289 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.713203 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:58.951327 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.951288 6999 generic.go:356] "Generic (PLEG): container finished" podID="701a7be4-531f-434e-9e1e-a85db672c13f" containerID="806a2456ccab2a55ffa9b335b6c11aa96f351de2e665a8a9023502ac4debe58f" exitCode=0 Oct 30 06:59:58.951764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.951345 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" Oct 30 06:59:58.951764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.951374 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" event={"ID":"701a7be4-531f-434e-9e1e-a85db672c13f","Type":"ContainerDied","Data":"806a2456ccab2a55ffa9b335b6c11aa96f351de2e665a8a9023502ac4debe58f"} Oct 30 06:59:58.951764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.951428 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85" event={"ID":"701a7be4-531f-434e-9e1e-a85db672c13f","Type":"ContainerDied","Data":"a35f91f3cd586ee9811a5042afc15881fcfbcec6d7c9e34d2f4a636b88faba33"} Oct 30 06:59:58.951764 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.951450 6999 scope.go:122] "RemoveContainer" containerID="806a2456ccab2a55ffa9b335b6c11aa96f351de2e665a8a9023502ac4debe58f" Oct 30 06:59:58.961447 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.961429 6999 scope.go:122] "RemoveContainer" containerID="806a2456ccab2a55ffa9b335b6c11aa96f351de2e665a8a9023502ac4debe58f" Oct 30 06:59:58.961737 ip-10-0-131-226 kubenswrapper[6999]: E1030 06:59:58.961717 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806a2456ccab2a55ffa9b335b6c11aa96f351de2e665a8a9023502ac4debe58f\": container with ID starting with 806a2456ccab2a55ffa9b335b6c11aa96f351de2e665a8a9023502ac4debe58f not found: ID does not exist" containerID="806a2456ccab2a55ffa9b335b6c11aa96f351de2e665a8a9023502ac4debe58f" Oct 30 06:59:58.961806 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:58.961746 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806a2456ccab2a55ffa9b335b6c11aa96f351de2e665a8a9023502ac4debe58f"} err="failed to get container status \"806a2456ccab2a55ffa9b335b6c11aa96f351de2e665a8a9023502ac4debe58f\": rpc error: code = NotFound desc = could not find container \"806a2456ccab2a55ffa9b335b6c11aa96f351de2e665a8a9023502ac4debe58f\": container with ID starting with 806a2456ccab2a55ffa9b335b6c11aa96f351de2e665a8a9023502ac4debe58f not found: ID does not exist" Oct 30 06:59:59.006652 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:59.006602 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85"] Oct 30 06:59:59.018469 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:59.018449 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-868b1f-6876c66d6f-dpk85"] Oct 30 06:59:59.168966 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:59.168937 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:59.415663 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:59.415627 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 06:59:59.608697 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:59.608667 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="701a7be4-531f-434e-9e1e-a85db672c13f" path="/var/lib/kubelet/pods/701a7be4-531f-434e-9e1e-a85db672c13f/volumes" Oct 30 06:59:59.853591 ip-10-0-131-226 kubenswrapper[6999]: I1030 06:59:59.853509 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:00.151877 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.151842 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf"] Oct 30 07:00:00.152328 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152313 6999 cpu_manager.go:414] "containerMap: removing container" podUID="784f9ba2-3d6b-4cf4-8546-32c975d271ed" containerName="main" Oct 30 07:00:00.152328 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152329 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="784f9ba2-3d6b-4cf4-8546-32c975d271ed" containerName="main" Oct 30 07:00:00.152435 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152344 6999 cpu_manager.go:414] "containerMap: removing container" podUID="776861d0-f966-44ca-b0eb-22b1a1b47e79" containerName="main" Oct 30 07:00:00.152435 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152351 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776861d0-f966-44ca-b0eb-22b1a1b47e79" containerName="main" Oct 30 07:00:00.152435 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152362 6999 cpu_manager.go:414] "containerMap: removing container" podUID="3162d9da-8ad6-4272-8374-af59367f2884" containerName="maas-api" Oct 30 07:00:00.152435 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152367 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3162d9da-8ad6-4272-8374-af59367f2884" containerName="maas-api" Oct 30 07:00:00.152435 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152373 6999 cpu_manager.go:414] "containerMap: removing container" podUID="4f4c6e9e-70db-47c3-82f5-e1798c4e6009" containerName="maas-api" Oct 30 07:00:00.152435 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152377 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f4c6e9e-70db-47c3-82f5-e1798c4e6009" containerName="maas-api" Oct 30 07:00:00.152435 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152388 6999 cpu_manager.go:414] "containerMap: removing container" podUID="1e88bddc-e0c1-4516-bd47-356b38380639" containerName="maas-api" Oct 30 07:00:00.152435 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152395 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e88bddc-e0c1-4516-bd47-356b38380639" containerName="maas-api" Oct 30 07:00:00.152435 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152409 6999 cpu_manager.go:414] "containerMap: removing container" podUID="f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" containerName="istio-proxy" Oct 30 07:00:00.152435 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152416 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" containerName="istio-proxy" Oct 30 07:00:00.152435 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152431 6999 cpu_manager.go:414] "containerMap: removing container" podUID="e4de3e31-9dff-43fc-8553-f859e41a1583" containerName="payload-processing" Oct 30 07:00:00.152435 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152438 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4de3e31-9dff-43fc-8553-f859e41a1583" containerName="payload-processing" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152451 6999 cpu_manager.go:414] "containerMap: removing container" podUID="9cf76c9b-fa0c-4575-aeae-c14e7487809b" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152459 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cf76c9b-fa0c-4575-aeae-c14e7487809b" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152470 6999 cpu_manager.go:414] "containerMap: removing container" podUID="1c9f6978-1b17-4e4d-a28b-aed920888f5f" containerName="main" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152475 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9f6978-1b17-4e4d-a28b-aed920888f5f" containerName="main" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152483 6999 cpu_manager.go:414] "containerMap: removing container" podUID="70b7d49d-e508-4a13-bc59-06a17f7e17fe" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152488 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b7d49d-e508-4a13-bc59-06a17f7e17fe" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152496 6999 cpu_manager.go:414] "containerMap: removing container" podUID="db9631b2-8f37-4465-83e0-6c67404aeb5b" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152501 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db9631b2-8f37-4465-83e0-6c67404aeb5b" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152509 6999 cpu_manager.go:414] "containerMap: removing container" podUID="7dce711f-1134-46f0-ace9-ff3080839ede" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152513 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dce711f-1134-46f0-ace9-ff3080839ede" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152520 6999 cpu_manager.go:414] "containerMap: removing container" podUID="ead1e0b2-5bb1-451d-865d-2fc40e243987" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152525 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead1e0b2-5bb1-451d-865d-2fc40e243987" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152533 6999 cpu_manager.go:414] "containerMap: removing container" podUID="ded3226b-1e88-48ff-a599-268f411abc76" containerName="payload-pre-processing" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152537 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded3226b-1e88-48ff-a599-268f411abc76" containerName="payload-pre-processing" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152545 6999 cpu_manager.go:414] "containerMap: removing container" podUID="4c821205-038e-4dd0-b122-837935d53312" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152549 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c821205-038e-4dd0-b122-837935d53312" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152556 6999 cpu_manager.go:414] "containerMap: removing container" podUID="f85d5059-e45f-418d-ae2c-2712a3241086" containerName="istio-proxy" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152560 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85d5059-e45f-418d-ae2c-2712a3241086" containerName="istio-proxy" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152567 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" containerName="istio-proxy" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152572 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" containerName="istio-proxy" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152577 6999 cpu_manager.go:414] "containerMap: removing container" podUID="afadd843-8c76-4a9c-b193-90b1ff6560e7" containerName="main" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152581 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afadd843-8c76-4a9c-b193-90b1ff6560e7" containerName="main" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152589 6999 cpu_manager.go:414] "containerMap: removing container" podUID="2e6794b5-28bc-4b10-a0d4-c1f529a09908" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152594 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e6794b5-28bc-4b10-a0d4-c1f529a09908" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152599 6999 cpu_manager.go:414] "containerMap: removing container" podUID="f79fd615-06d2-4a53-85cc-0c7da6834671" containerName="main" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152604 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f79fd615-06d2-4a53-85cc-0c7da6834671" containerName="main" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152610 6999 cpu_manager.go:414] "containerMap: removing container" podUID="dd69357f-342b-4367-96c0-a98fa610742d" containerName="istio-proxy" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152614 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd69357f-342b-4367-96c0-a98fa610742d" containerName="istio-proxy" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152623 6999 cpu_manager.go:414] "containerMap: removing container" podUID="e88d9911-2196-41cd-a9ae-3e73aca123e5" containerName="istio-proxy" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152628 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e88d9911-2196-41cd-a9ae-3e73aca123e5" containerName="istio-proxy" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152636 6999 cpu_manager.go:414] "containerMap: removing container" podUID="5f2aec32-15b0-4a59-839b-a1e23f337319" containerName="istio-proxy" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152641 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2aec32-15b0-4a59-839b-a1e23f337319" containerName="istio-proxy" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152648 6999 cpu_manager.go:414] "containerMap: removing container" podUID="a19ca2c4-c2ed-4d38-9537-fc394357aa84" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152654 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a19ca2c4-c2ed-4d38-9537-fc394357aa84" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152669 6999 cpu_manager.go:414] "containerMap: removing container" podUID="0f526f75-f591-46d3-8bef-6d9c8b76ded6" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152677 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f526f75-f591-46d3-8bef-6d9c8b76ded6" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152685 6999 cpu_manager.go:414] "containerMap: removing container" podUID="7bc29aa6-9b21-4581-94f3-3e5302cc4b6b" containerName="payload-pre-processing" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152692 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc29aa6-9b21-4581-94f3-3e5302cc4b6b" containerName="payload-pre-processing" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152701 6999 cpu_manager.go:414] "containerMap: removing container" podUID="8111e1b2-a50e-4d15-8efb-00febd24b5e3" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152709 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8111e1b2-a50e-4d15-8efb-00febd24b5e3" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152718 6999 cpu_manager.go:414] "containerMap: removing container" podUID="3e6752f6-ee5f-4800-8fd8-b9df6aa69274" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152726 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6752f6-ee5f-4800-8fd8-b9df6aa69274" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152736 6999 cpu_manager.go:414] "containerMap: removing container" podUID="85d15937-bbc6-440d-ba22-f7989f1e774d" containerName="istio-proxy" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152743 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d15937-bbc6-440d-ba22-f7989f1e774d" containerName="istio-proxy" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152756 6999 cpu_manager.go:414] "containerMap: removing container" podUID="dceee649-894a-4330-a9df-0097ca9e352e" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152763 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dceee649-894a-4330-a9df-0097ca9e352e" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152777 6999 cpu_manager.go:414] "containerMap: removing container" podUID="7fc83c12-1924-49f9-9725-901610b7ed41" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152784 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc83c12-1924-49f9-9725-901610b7ed41" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152793 6999 cpu_manager.go:414] "containerMap: removing container" podUID="e7b70aac-3036-44cb-ae26-d4e0bea0a45e" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152800 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7b70aac-3036-44cb-ae26-d4e0bea0a45e" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152811 6999 cpu_manager.go:414] "containerMap: removing container" podUID="701a7be4-531f-434e-9e1e-a85db672c13f" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152817 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="701a7be4-531f-434e-9e1e-a85db672c13f" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152827 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b2701c51-6377-411c-9924-715ff0765564" containerName="main" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152833 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2701c51-6377-411c-9924-715ff0765564" containerName="main" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152845 6999 cpu_manager.go:414] "containerMap: removing container" podUID="fb73c18a-0b2c-44b4-a5de-133963b43a88" containerName="payload-pre-processing" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152853 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb73c18a-0b2c-44b4-a5de-133963b43a88" containerName="payload-pre-processing" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152864 6999 cpu_manager.go:414] "containerMap: removing container" podUID="e62ce8a0-4a42-4943-8ced-5f033661b9be" containerName="payload-processing" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152871 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e62ce8a0-4a42-4943-8ced-5f033661b9be" containerName="payload-processing" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152886 6999 cpu_manager.go:414] "containerMap: removing container" podUID="67c137ac-abb6-412a-9dac-7639c95abaa0" containerName="maas-api" Oct 30 07:00:00.152859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152894 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c137ac-abb6-412a-9dac-7639c95abaa0" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152905 6999 cpu_manager.go:414] "containerMap: removing container" podUID="79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" containerName="main" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152912 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" containerName="main" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152925 6999 cpu_manager.go:414] "containerMap: removing container" podUID="8e38cac7-3426-497c-8b84-e7552169bb7c" containerName="main" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152932 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e38cac7-3426-497c-8b84-e7552169bb7c" containerName="main" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152943 6999 cpu_manager.go:414] "containerMap: removing container" podUID="115bfbd1-fef2-4d48-a261-a204e607f077" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.152950 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115bfbd1-fef2-4d48-a261-a204e607f077" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153087 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="e62ce8a0-4a42-4943-8ced-5f033661b9be" containerName="payload-processing" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153103 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c137ac-abb6-412a-9dac-7639c95abaa0" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153111 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b8b3f2-6e6b-49cb-a973-8fb098ff3bc4" containerName="main" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153117 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e38cac7-3426-497c-8b84-e7552169bb7c" containerName="main" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153122 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="115bfbd1-fef2-4d48-a261-a204e607f077" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153131 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="784f9ba2-3d6b-4cf4-8546-32c975d271ed" containerName="main" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153139 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="776861d0-f966-44ca-b0eb-22b1a1b47e79" containerName="main" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153149 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="3162d9da-8ad6-4272-8374-af59367f2884" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153163 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f4c6e9e-70db-47c3-82f5-e1798c4e6009" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153171 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e88bddc-e0c1-4516-bd47-356b38380639" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153180 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d7acbc-310d-4ac4-ba1f-1f60fdcb95fb" containerName="istio-proxy" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153191 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4de3e31-9dff-43fc-8553-f859e41a1583" containerName="payload-processing" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153199 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cf76c9b-fa0c-4575-aeae-c14e7487809b" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153281 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c9f6978-1b17-4e4d-a28b-aed920888f5f" containerName="main" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153295 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b7d49d-e508-4a13-bc59-06a17f7e17fe" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153307 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="db9631b2-8f37-4465-83e0-6c67404aeb5b" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153317 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dce711f-1134-46f0-ace9-ff3080839ede" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153327 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead1e0b2-5bb1-451d-865d-2fc40e243987" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153336 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded3226b-1e88-48ff-a599-268f411abc76" containerName="payload-pre-processing" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153345 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c821205-038e-4dd0-b122-837935d53312" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153352 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85d5059-e45f-418d-ae2c-2712a3241086" containerName="istio-proxy" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153361 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30fe423-3a43-4f07-aa4a-f147ea8f7cd9" containerName="istio-proxy" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153375 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="afadd843-8c76-4a9c-b193-90b1ff6560e7" containerName="main" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153381 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e6794b5-28bc-4b10-a0d4-c1f529a09908" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153386 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="f79fd615-06d2-4a53-85cc-0c7da6834671" containerName="main" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153393 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd69357f-342b-4367-96c0-a98fa610742d" containerName="istio-proxy" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153399 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="e88d9911-2196-41cd-a9ae-3e73aca123e5" containerName="istio-proxy" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153406 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2aec32-15b0-4a59-839b-a1e23f337319" containerName="istio-proxy" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153411 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="a19ca2c4-c2ed-4d38-9537-fc394357aa84" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153417 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f526f75-f591-46d3-8bef-6d9c8b76ded6" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153422 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc29aa6-9b21-4581-94f3-3e5302cc4b6b" containerName="payload-pre-processing" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153428 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="8111e1b2-a50e-4d15-8efb-00febd24b5e3" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153434 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6752f6-ee5f-4800-8fd8-b9df6aa69274" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153439 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d15937-bbc6-440d-ba22-f7989f1e774d" containerName="istio-proxy" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153444 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="dceee649-894a-4330-a9df-0097ca9e352e" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153451 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc83c12-1924-49f9-9725-901610b7ed41" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153456 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7b70aac-3036-44cb-ae26-d4e0bea0a45e" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153462 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="701a7be4-531f-434e-9e1e-a85db672c13f" containerName="maas-api" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153468 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2701c51-6377-411c-9924-715ff0765564" containerName="main" Oct 30 07:00:00.155074 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.153475 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb73c18a-0b2c-44b4-a5de-133963b43a88" containerName="payload-pre-processing" Oct 30 07:00:00.158293 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.158276 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs"] Oct 30 07:00:00.158412 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.158394 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" Oct 30 07:00:00.161958 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.161937 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" Oct 30 07:00:00.178622 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.178595 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf"] Oct 30 07:00:00.188610 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.188587 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 07:00:00.191262 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.190557 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs"] Oct 30 07:00:00.208908 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.208882 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qhcm\" (UniqueName: \"kubernetes.io/projected/b5b8d268-3b87-4153-9759-a902e5dd3151-kube-api-access-8qhcm\") pod \"maas-api-key-cleanup-29889060-fffzs\" (UID: \"b5b8d268-3b87-4153-9759-a902e5dd3151\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" Oct 30 07:00:00.209033 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.208933 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55jqk\" (UniqueName: \"kubernetes.io/projected/f467d292-2ba6-4287-be24-9016afdc01b0-kube-api-access-55jqk\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf\" (UID: \"f467d292-2ba6-4287-be24-9016afdc01b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" Oct 30 07:00:00.209142 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.209115 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b8d268-3b87-4153-9759-a902e5dd3151-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889060-fffzs\" (UID: \"b5b8d268-3b87-4153-9759-a902e5dd3151\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" Oct 30 07:00:00.209275 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.209256 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f467d292-2ba6-4287-be24-9016afdc01b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf\" (UID: \"f467d292-2ba6-4287-be24-9016afdc01b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" Oct 30 07:00:00.310676 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.310630 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b8d268-3b87-4153-9759-a902e5dd3151-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889060-fffzs\" (UID: \"b5b8d268-3b87-4153-9759-a902e5dd3151\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" Oct 30 07:00:00.310862 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.310711 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f467d292-2ba6-4287-be24-9016afdc01b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf\" (UID: \"f467d292-2ba6-4287-be24-9016afdc01b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" Oct 30 07:00:00.310862 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.310817 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qhcm\" (UniqueName: \"kubernetes.io/projected/b5b8d268-3b87-4153-9759-a902e5dd3151-kube-api-access-8qhcm\") pod \"maas-api-key-cleanup-29889060-fffzs\" (UID: \"b5b8d268-3b87-4153-9759-a902e5dd3151\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" Oct 30 07:00:00.310862 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.310854 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55jqk\" (UniqueName: \"kubernetes.io/projected/f467d292-2ba6-4287-be24-9016afdc01b0-kube-api-access-55jqk\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf\" (UID: \"f467d292-2ba6-4287-be24-9016afdc01b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" Oct 30 07:00:00.311494 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.311466 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f467d292-2ba6-4287-be24-9016afdc01b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf\" (UID: \"f467d292-2ba6-4287-be24-9016afdc01b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" Oct 30 07:00:00.311614 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.311494 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b8d268-3b87-4153-9759-a902e5dd3151-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889060-fffzs\" (UID: \"b5b8d268-3b87-4153-9759-a902e5dd3151\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" Oct 30 07:00:00.334367 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.334336 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qhcm\" (UniqueName: \"kubernetes.io/projected/b5b8d268-3b87-4153-9759-a902e5dd3151-kube-api-access-8qhcm\") pod \"maas-api-key-cleanup-29889060-fffzs\" (UID: \"b5b8d268-3b87-4153-9759-a902e5dd3151\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" Oct 30 07:00:00.334473 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.334344 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55jqk\" (UniqueName: \"kubernetes.io/projected/f467d292-2ba6-4287-be24-9016afdc01b0-kube-api-access-55jqk\") pod \"maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf\" (UID: \"f467d292-2ba6-4287-be24-9016afdc01b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" Oct 30 07:00:00.474913 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.474828 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" Oct 30 07:00:00.487605 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.486512 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" Oct 30 07:00:00.661413 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.661379 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:00.781184 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.781160 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf"] Oct 30 07:00:00.783523 ip-10-0-131-226 kubenswrapper[6999]: W1030 07:00:00.783499 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf467d292_2ba6_4287_be24_9016afdc01b0.slice/crio-10b83d8dbd1bea7f1848affdfd4745226f40df80f972241bcad222cc19897406 WatchSource:0}: Error finding container 10b83d8dbd1bea7f1848affdfd4745226f40df80f972241bcad222cc19897406: Status 404 returned error can't find the container with id 10b83d8dbd1bea7f1848affdfd4745226f40df80f972241bcad222cc19897406 Oct 30 07:00:00.796456 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.796430 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs"] Oct 30 07:00:00.798939 ip-10-0-131-226 kubenswrapper[6999]: W1030 07:00:00.798875 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b8d268_3b87_4153_9759_a902e5dd3151.slice/crio-6e0793f14eefcd6a6aa6f243b55b18dc140b78962c03ac22cf7d613c3bb2ce64 WatchSource:0}: Error finding container 6e0793f14eefcd6a6aa6f243b55b18dc140b78962c03ac22cf7d613c3bb2ce64: Status 404 returned error can't find the container with id 6e0793f14eefcd6a6aa6f243b55b18dc140b78962c03ac22cf7d613c3bb2ce64 Oct 30 07:00:00.962316 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.962284 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" event={"ID":"b5b8d268-3b87-4153-9759-a902e5dd3151","Type":"ContainerStarted","Data":"6e0793f14eefcd6a6aa6f243b55b18dc140b78962c03ac22cf7d613c3bb2ce64"} Oct 30 07:00:00.963514 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:00.963492 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" event={"ID":"f467d292-2ba6-4287-be24-9016afdc01b0","Type":"ContainerStarted","Data":"10b83d8dbd1bea7f1848affdfd4745226f40df80f972241bcad222cc19897406"} Oct 30 07:00:01.970711 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:01.970676 6999 generic.go:356] "Generic (PLEG): container finished" podID="f467d292-2ba6-4287-be24-9016afdc01b0" containerID="300a8ab476de404762913638f9b9461f2b18bc0098740b45c45d7a5c7008b93f" exitCode=60 Oct 30 07:00:01.971285 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:01.970773 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" event={"ID":"f467d292-2ba6-4287-be24-9016afdc01b0","Type":"ContainerDied","Data":"300a8ab476de404762913638f9b9461f2b18bc0098740b45c45d7a5c7008b93f"} Oct 30 07:00:01.971285 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:01.971057 6999 scope.go:122] "RemoveContainer" containerID="300a8ab476de404762913638f9b9461f2b18bc0098740b45c45d7a5c7008b93f" Oct 30 07:00:01.972645 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:01.972617 6999 generic.go:356] "Generic (PLEG): container finished" podID="b5b8d268-3b87-4153-9759-a902e5dd3151" containerID="a5b433a78bfbc4680d18788e5bfe3f5615aad50a3d6c03a1499d0d31ea3d08a5" exitCode=60 Oct 30 07:00:01.972788 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:01.972684 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" event={"ID":"b5b8d268-3b87-4153-9759-a902e5dd3151","Type":"ContainerDied","Data":"a5b433a78bfbc4680d18788e5bfe3f5615aad50a3d6c03a1499d0d31ea3d08a5"} Oct 30 07:00:01.972939 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:01.972924 6999 scope.go:122] "RemoveContainer" containerID="a5b433a78bfbc4680d18788e5bfe3f5615aad50a3d6c03a1499d0d31ea3d08a5" Oct 30 07:00:02.111664 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.111621 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx"] Oct 30 07:00:02.111885 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.111855 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" podUID="51d5ac7b-9102-4ecb-bb87-53182e8898d2" containerName="maas-api" containerID="cri-o://fab04c8149922bdb44bea2840b56ecfabd540a2685e10641cdea7d745f2e28ce" gracePeriod=30 Oct 30 07:00:02.166984 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.166951 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:02.317829 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.317775 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:02.443702 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.443680 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 07:00:02.459349 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.459322 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:02.534460 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.534369 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d5ac7b-9102-4ecb-bb87-53182e8898d2-kube-api-access-c2mz4\" (UniqueName: \"kubernetes.io/projected/51d5ac7b-9102-4ecb-bb87-53182e8898d2-kube-api-access-c2mz4\") pod \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " Oct 30 07:00:02.534626 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.534503 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d5ac7b-9102-4ecb-bb87-53182e8898d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d5ac7b-9102-4ecb-bb87-53182e8898d2-openshift-service-ca\") pod \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " Oct 30 07:00:02.534708 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.534638 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51d5ac7b-9102-4ecb-bb87-53182e8898d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/51d5ac7b-9102-4ecb-bb87-53182e8898d2-metrics-certs\") pod \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " Oct 30 07:00:02.534708 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.534670 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51d5ac7b-9102-4ecb-bb87-53182e8898d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51d5ac7b-9102-4ecb-bb87-53182e8898d2-maas-api-tls\") pod \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\" (UID: \"51d5ac7b-9102-4ecb-bb87-53182e8898d2\") " Oct 30 07:00:02.534904 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.534878 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d5ac7b-9102-4ecb-bb87-53182e8898d2-openshift-service-ca" pod "51d5ac7b-9102-4ecb-bb87-53182e8898d2" (UID: "51d5ac7b-9102-4ecb-bb87-53182e8898d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:02.535047 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.535029 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/51d5ac7b-9102-4ecb-bb87-53182e8898d2-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:02.536888 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.536863 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d5ac7b-9102-4ecb-bb87-53182e8898d2-kube-api-access-c2mz4" pod "51d5ac7b-9102-4ecb-bb87-53182e8898d2" (UID: "51d5ac7b-9102-4ecb-bb87-53182e8898d2"). InnerVolumeSpecName "kube-api-access-c2mz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:02.537031 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.536981 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51d5ac7b-9102-4ecb-bb87-53182e8898d2-maas-api-tls" pod "51d5ac7b-9102-4ecb-bb87-53182e8898d2" (UID: "51d5ac7b-9102-4ecb-bb87-53182e8898d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:02.537314 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.537299 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51d5ac7b-9102-4ecb-bb87-53182e8898d2-metrics-certs" pod "51d5ac7b-9102-4ecb-bb87-53182e8898d2" (UID: "51d5ac7b-9102-4ecb-bb87-53182e8898d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:02.596568 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.596536 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:02.636719 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.636678 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/51d5ac7b-9102-4ecb-bb87-53182e8898d2-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:02.636719 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.636717 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/51d5ac7b-9102-4ecb-bb87-53182e8898d2-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:02.636973 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.636734 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2mz4\" (UniqueName: \"kubernetes.io/projected/51d5ac7b-9102-4ecb-bb87-53182e8898d2-kube-api-access-c2mz4\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:02.810997 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.810911 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:02.850405 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.850355 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" podUID="fc8e33ed-9831-4c77-8fdd-b64f9a593b99" containerName="payload-pre-processing" containerID="cri-o://88d16e753a568ded154fb0bce97634a071c074a264845c5f6033c3220bc92eb4" gracePeriod=25 Oct 30 07:00:02.954497 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.954471 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:02.980024 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.979994 6999 generic.go:356] "Generic (PLEG): container finished" podID="51d5ac7b-9102-4ecb-bb87-53182e8898d2" containerID="fab04c8149922bdb44bea2840b56ecfabd540a2685e10641cdea7d745f2e28ce" exitCode=0 Oct 30 07:00:02.980484 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.980049 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" Oct 30 07:00:02.980484 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.980068 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" event={"ID":"51d5ac7b-9102-4ecb-bb87-53182e8898d2","Type":"ContainerDied","Data":"fab04c8149922bdb44bea2840b56ecfabd540a2685e10641cdea7d745f2e28ce"} Oct 30 07:00:02.980484 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.980113 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx" event={"ID":"51d5ac7b-9102-4ecb-bb87-53182e8898d2","Type":"ContainerDied","Data":"43c3d5377b18fbf748f7308be4e7d31c5eded130a35608124ea4c250f5fcb95d"} Oct 30 07:00:02.980484 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.980141 6999 scope.go:122] "RemoveContainer" containerID="fab04c8149922bdb44bea2840b56ecfabd540a2685e10641cdea7d745f2e28ce" Oct 30 07:00:02.982295 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.982175 6999 generic.go:356] "Generic (PLEG): container finished" podID="b5b8d268-3b87-4153-9759-a902e5dd3151" containerID="c2173f60acd0c4f9573084edee454f1f9029b7adbcbeafceea82a24ef9210578" exitCode=60 Oct 30 07:00:02.982295 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.982254 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" event={"ID":"b5b8d268-3b87-4153-9759-a902e5dd3151","Type":"ContainerDied","Data":"c2173f60acd0c4f9573084edee454f1f9029b7adbcbeafceea82a24ef9210578"} Oct 30 07:00:02.982590 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.982567 6999 scope.go:122] "RemoveContainer" containerID="c2173f60acd0c4f9573084edee454f1f9029b7adbcbeafceea82a24ef9210578" Oct 30 07:00:02.982842 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:00:02.982819 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889060-fffzs_odh-ai-gateway-infra(b5b8d268-3b87-4153-9759-a902e5dd3151)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" podUID="b5b8d268-3b87-4153-9759-a902e5dd3151" Oct 30 07:00:02.984386 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.984355 6999 generic.go:356] "Generic (PLEG): container finished" podID="f467d292-2ba6-4287-be24-9016afdc01b0" containerID="c535fccbd8379ea94468d87c1c221d041dab588e529ab16d8f446620067b29bf" exitCode=60 Oct 30 07:00:02.984477 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.984431 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" event={"ID":"f467d292-2ba6-4287-be24-9016afdc01b0","Type":"ContainerDied","Data":"c535fccbd8379ea94468d87c1c221d041dab588e529ab16d8f446620067b29bf"} Oct 30 07:00:02.984649 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.984631 6999 scope.go:122] "RemoveContainer" containerID="c535fccbd8379ea94468d87c1c221d041dab588e529ab16d8f446620067b29bf" Oct 30 07:00:02.984911 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:00:02.984887 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf_odh-ai-gateway-infra(f467d292-2ba6-4287-be24-9016afdc01b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" podUID="f467d292-2ba6-4287-be24-9016afdc01b0" Oct 30 07:00:02.995452 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.995434 6999 scope.go:122] "RemoveContainer" containerID="fab04c8149922bdb44bea2840b56ecfabd540a2685e10641cdea7d745f2e28ce" Oct 30 07:00:02.995720 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:00:02.995702 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab04c8149922bdb44bea2840b56ecfabd540a2685e10641cdea7d745f2e28ce\": container with ID starting with fab04c8149922bdb44bea2840b56ecfabd540a2685e10641cdea7d745f2e28ce not found: ID does not exist" containerID="fab04c8149922bdb44bea2840b56ecfabd540a2685e10641cdea7d745f2e28ce" Oct 30 07:00:02.995760 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.995728 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab04c8149922bdb44bea2840b56ecfabd540a2685e10641cdea7d745f2e28ce"} err="failed to get container status \"fab04c8149922bdb44bea2840b56ecfabd540a2685e10641cdea7d745f2e28ce\": rpc error: code = NotFound desc = could not find container \"fab04c8149922bdb44bea2840b56ecfabd540a2685e10641cdea7d745f2e28ce\": container with ID starting with fab04c8149922bdb44bea2840b56ecfabd540a2685e10641cdea7d745f2e28ce not found: ID does not exist" Oct 30 07:00:02.995760 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:02.995754 6999 scope.go:122] "RemoveContainer" containerID="a5b433a78bfbc4680d18788e5bfe3f5615aad50a3d6c03a1499d0d31ea3d08a5" Oct 30 07:00:03.008349 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.008332 6999 scope.go:122] "RemoveContainer" containerID="300a8ab476de404762913638f9b9461f2b18bc0098740b45c45d7a5c7008b93f" Oct 30 07:00:03.072192 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.072118 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx"] Oct 30 07:00:03.076078 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.076056 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-30fbb6-77bc8f86b4-mrrfx"] Oct 30 07:00:03.402043 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.401968 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:03.546600 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.546567 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:03.607926 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.607848 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d5ac7b-9102-4ecb-bb87-53182e8898d2" path="/var/lib/kubelet/pods/51d5ac7b-9102-4ecb-bb87-53182e8898d2/volumes" Oct 30 07:00:03.669482 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.669450 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl"] Oct 30 07:00:03.670039 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.670018 6999 cpu_manager.go:414] "containerMap: removing container" podUID="51d5ac7b-9102-4ecb-bb87-53182e8898d2" containerName="maas-api" Oct 30 07:00:03.670039 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.670037 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d5ac7b-9102-4ecb-bb87-53182e8898d2" containerName="maas-api" Oct 30 07:00:03.670177 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.670136 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d5ac7b-9102-4ecb-bb87-53182e8898d2" containerName="maas-api" Oct 30 07:00:03.674663 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.674646 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" Oct 30 07:00:03.690913 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.690884 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:00:03.693245 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.693201 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl"] Oct 30 07:00:03.751314 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.751280 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqmdm\" (UniqueName: \"kubernetes.io/projected/11cdb156-da38-4a6a-8976-f0a1a2eff985-kube-api-access-jqmdm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl\" (UID: \"11cdb156-da38-4a6a-8976-f0a1a2eff985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" Oct 30 07:00:03.751504 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.751446 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11cdb156-da38-4a6a-8976-f0a1a2eff985-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl\" (UID: \"11cdb156-da38-4a6a-8976-f0a1a2eff985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" Oct 30 07:00:03.751504 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.751489 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11cdb156-da38-4a6a-8976-f0a1a2eff985-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl\" (UID: \"11cdb156-da38-4a6a-8976-f0a1a2eff985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" Oct 30 07:00:03.852282 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.852238 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11cdb156-da38-4a6a-8976-f0a1a2eff985-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl\" (UID: \"11cdb156-da38-4a6a-8976-f0a1a2eff985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" Oct 30 07:00:03.852282 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.852289 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqmdm\" (UniqueName: \"kubernetes.io/projected/11cdb156-da38-4a6a-8976-f0a1a2eff985-kube-api-access-jqmdm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl\" (UID: \"11cdb156-da38-4a6a-8976-f0a1a2eff985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" Oct 30 07:00:03.852500 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.852379 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11cdb156-da38-4a6a-8976-f0a1a2eff985-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl\" (UID: \"11cdb156-da38-4a6a-8976-f0a1a2eff985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" Oct 30 07:00:03.853019 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.852993 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11cdb156-da38-4a6a-8976-f0a1a2eff985-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl\" (UID: \"11cdb156-da38-4a6a-8976-f0a1a2eff985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" Oct 30 07:00:03.854930 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.854905 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11cdb156-da38-4a6a-8976-f0a1a2eff985-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl\" (UID: \"11cdb156-da38-4a6a-8976-f0a1a2eff985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" Oct 30 07:00:03.876044 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.876013 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqmdm\" (UniqueName: \"kubernetes.io/projected/11cdb156-da38-4a6a-8976-f0a1a2eff985-kube-api-access-jqmdm\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl\" (UID: \"11cdb156-da38-4a6a-8976-f0a1a2eff985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" Oct 30 07:00:03.988323 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.988293 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" Oct 30 07:00:03.992868 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.992838 6999 scope.go:122] "RemoveContainer" containerID="c2173f60acd0c4f9573084edee454f1f9029b7adbcbeafceea82a24ef9210578" Oct 30 07:00:03.993154 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:00:03.993120 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889060-fffzs_odh-ai-gateway-infra(b5b8d268-3b87-4153-9759-a902e5dd3151)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" podUID="b5b8d268-3b87-4153-9759-a902e5dd3151" Oct 30 07:00:03.994698 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:03.994677 6999 scope.go:122] "RemoveContainer" containerID="c535fccbd8379ea94468d87c1c221d041dab588e529ab16d8f446620067b29bf" Oct 30 07:00:03.994893 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:00:03.994876 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf_odh-ai-gateway-infra(f467d292-2ba6-4287-be24-9016afdc01b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" podUID="f467d292-2ba6-4287-be24-9016afdc01b0" Oct 30 07:00:04.083159 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:04.083121 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:04.145422 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:00:04.145325 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 07:00:04.292039 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:04.292014 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl"] Oct 30 07:00:04.294190 ip-10-0-131-226 kubenswrapper[6999]: W1030 07:00:04.294165 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cdb156_da38_4a6a_8976_f0a1a2eff985.slice/crio-2fb9874b3e7044bdb60f2891bba1c0037521f5a6093eb0262b670b32af910fba WatchSource:0}: Error finding container 2fb9874b3e7044bdb60f2891bba1c0037521f5a6093eb0262b670b32af910fba: Status 404 returned error can't find the container with id 2fb9874b3e7044bdb60f2891bba1c0037521f5a6093eb0262b670b32af910fba Oct 30 07:00:04.888550 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:04.888513 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:05.002024 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:05.001977 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" event={"ID":"11cdb156-da38-4a6a-8976-f0a1a2eff985","Type":"ContainerStarted","Data":"bc4540baa7d99bce996ded2e8892c0677fd6cf6b8f73c209debfbeddc4bc1618"} Oct 30 07:00:05.002543 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:05.002043 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" event={"ID":"11cdb156-da38-4a6a-8976-f0a1a2eff985","Type":"ContainerStarted","Data":"2fb9874b3e7044bdb60f2891bba1c0037521f5a6093eb0262b670b32af910fba"} Oct 30 07:00:05.045072 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:05.044998 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" podStartSLOduration=2.044978781 podStartE2EDuration="2.044978781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:05.043875841 +0000 UTC m=+3971.552895740" watchObservedRunningTime="2026-10-30 07:00:05.044978781 +0000 UTC m=+3971.553998668" Oct 30 07:00:06.009989 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:06.009953 6999 generic.go:356] "Generic (PLEG): container finished" podID="11cdb156-da38-4a6a-8976-f0a1a2eff985" containerID="bc4540baa7d99bce996ded2e8892c0677fd6cf6b8f73c209debfbeddc4bc1618" exitCode=0 Oct 30 07:00:06.010485 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:06.010052 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" event={"ID":"11cdb156-da38-4a6a-8976-f0a1a2eff985","Type":"ContainerDied","Data":"bc4540baa7d99bce996ded2e8892c0677fd6cf6b8f73c209debfbeddc4bc1618"} Oct 30 07:00:06.803543 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:06.803500 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:06.945594 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:06.945557 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:07.151458 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:07.151429 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:07.162039 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:07.162020 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" Oct 30 07:00:07.291453 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:07.291415 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11cdb156-da38-4a6a-8976-f0a1a2eff985-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11cdb156-da38-4a6a-8976-f0a1a2eff985-maas-api-openshift-ca\") pod \"11cdb156-da38-4a6a-8976-f0a1a2eff985\" (UID: \"11cdb156-da38-4a6a-8976-f0a1a2eff985\") " Oct 30 07:00:07.291628 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:07.291547 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11cdb156-da38-4a6a-8976-f0a1a2eff985-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11cdb156-da38-4a6a-8976-f0a1a2eff985-maas-api-xks-ca\") pod \"11cdb156-da38-4a6a-8976-f0a1a2eff985\" (UID: \"11cdb156-da38-4a6a-8976-f0a1a2eff985\") " Oct 30 07:00:07.291628 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:07.291585 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11cdb156-da38-4a6a-8976-f0a1a2eff985-kube-api-access-jqmdm\" (UniqueName: \"kubernetes.io/projected/11cdb156-da38-4a6a-8976-f0a1a2eff985-kube-api-access-jqmdm\") pod \"11cdb156-da38-4a6a-8976-f0a1a2eff985\" (UID: \"11cdb156-da38-4a6a-8976-f0a1a2eff985\") " Oct 30 07:00:07.291763 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:07.291699 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cdb156-da38-4a6a-8976-f0a1a2eff985-maas-api-openshift-ca" pod "11cdb156-da38-4a6a-8976-f0a1a2eff985" (UID: "11cdb156-da38-4a6a-8976-f0a1a2eff985"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:07.291902 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:07.291884 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11cdb156-da38-4a6a-8976-f0a1a2eff985-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:07.293779 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:07.293756 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11cdb156-da38-4a6a-8976-f0a1a2eff985-maas-api-xks-ca" pod "11cdb156-da38-4a6a-8976-f0a1a2eff985" (UID: "11cdb156-da38-4a6a-8976-f0a1a2eff985"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:07.293887 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:07.293808 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11cdb156-da38-4a6a-8976-f0a1a2eff985-kube-api-access-jqmdm" pod "11cdb156-da38-4a6a-8976-f0a1a2eff985" (UID: "11cdb156-da38-4a6a-8976-f0a1a2eff985"). InnerVolumeSpecName "kube-api-access-jqmdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:07.310305 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:07.310284 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:07.393414 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:07.393386 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11cdb156-da38-4a6a-8976-f0a1a2eff985-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:07.393543 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:07.393419 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqmdm\" (UniqueName: \"kubernetes.io/projected/11cdb156-da38-4a6a-8976-f0a1a2eff985-kube-api-access-jqmdm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:07.534363 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:07.534331 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:08.022394 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.022358 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" Oct 30 07:00:08.022394 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.022388 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl" event={"ID":"11cdb156-da38-4a6a-8976-f0a1a2eff985","Type":"ContainerDied","Data":"2fb9874b3e7044bdb60f2891bba1c0037521f5a6093eb0262b670b32af910fba"} Oct 30 07:00:08.022616 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.022436 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fb9874b3e7044bdb60f2891bba1c0037521f5a6093eb0262b670b32af910fba" Oct 30 07:00:08.085574 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.085542 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:08.323791 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.323718 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj"] Oct 30 07:00:08.324241 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.324200 6999 cpu_manager.go:414] "containerMap: removing container" podUID="11cdb156-da38-4a6a-8976-f0a1a2eff985" containerName="revoke-keys" Oct 30 07:00:08.324241 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.324233 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11cdb156-da38-4a6a-8976-f0a1a2eff985" containerName="revoke-keys" Oct 30 07:00:08.324366 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.324299 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="11cdb156-da38-4a6a-8976-f0a1a2eff985" containerName="revoke-keys" Oct 30 07:00:08.327912 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.327892 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" Oct 30 07:00:08.343278 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.343192 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:00:08.348266 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.348244 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj"] Oct 30 07:00:08.402866 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.402837 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/108a685d-1c86-402f-914f-d7574a1deef5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj\" (UID: \"108a685d-1c86-402f-914f-d7574a1deef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" Oct 30 07:00:08.403013 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.402886 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/108a685d-1c86-402f-914f-d7574a1deef5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj\" (UID: \"108a685d-1c86-402f-914f-d7574a1deef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" Oct 30 07:00:08.403057 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.403006 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr27b\" (UniqueName: \"kubernetes.io/projected/108a685d-1c86-402f-914f-d7574a1deef5-kube-api-access-pr27b\") pod \"maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj\" (UID: \"108a685d-1c86-402f-914f-d7574a1deef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" Oct 30 07:00:08.503707 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.503674 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr27b\" (UniqueName: \"kubernetes.io/projected/108a685d-1c86-402f-914f-d7574a1deef5-kube-api-access-pr27b\") pod \"maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj\" (UID: \"108a685d-1c86-402f-914f-d7574a1deef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" Oct 30 07:00:08.503862 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.503767 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/108a685d-1c86-402f-914f-d7574a1deef5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj\" (UID: \"108a685d-1c86-402f-914f-d7574a1deef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" Oct 30 07:00:08.503862 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.503802 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/108a685d-1c86-402f-914f-d7574a1deef5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj\" (UID: \"108a685d-1c86-402f-914f-d7574a1deef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" Oct 30 07:00:08.504490 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.504464 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/108a685d-1c86-402f-914f-d7574a1deef5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj\" (UID: \"108a685d-1c86-402f-914f-d7574a1deef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" Oct 30 07:00:08.506096 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.506073 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/108a685d-1c86-402f-914f-d7574a1deef5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj\" (UID: \"108a685d-1c86-402f-914f-d7574a1deef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" Oct 30 07:00:08.527496 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.527469 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr27b\" (UniqueName: \"kubernetes.io/projected/108a685d-1c86-402f-914f-d7574a1deef5-kube-api-access-pr27b\") pod \"maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj\" (UID: \"108a685d-1c86-402f-914f-d7574a1deef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" Oct 30 07:00:08.641879 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.641853 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" Oct 30 07:00:08.841428 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.841402 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:08.942102 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:08.942079 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj"] Oct 30 07:00:08.944793 ip-10-0-131-226 kubenswrapper[6999]: W1030 07:00:08.944764 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108a685d_1c86_402f_914f_d7574a1deef5.slice/crio-4c00002f05fc55748fed9fdb8e411abfc756e08bcec2aa2c6996736047a4c46f WatchSource:0}: Error finding container 4c00002f05fc55748fed9fdb8e411abfc756e08bcec2aa2c6996736047a4c46f: Status 404 returned error can't find the container with id 4c00002f05fc55748fed9fdb8e411abfc756e08bcec2aa2c6996736047a4c46f Oct 30 07:00:09.029139 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:09.029113 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" event={"ID":"108a685d-1c86-402f-914f-d7574a1deef5","Type":"ContainerStarted","Data":"4c00002f05fc55748fed9fdb8e411abfc756e08bcec2aa2c6996736047a4c46f"} Oct 30 07:00:09.296859 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:09.296829 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:10.036106 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:10.036026 6999 generic.go:356] "Generic (PLEG): container finished" podID="108a685d-1c86-402f-914f-d7574a1deef5" containerID="8daa1a07168f774b19f950db44ae3bd571b85182d5607f79915383f694c1a20d" exitCode=0 Oct 30 07:00:10.036522 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:10.036112 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" event={"ID":"108a685d-1c86-402f-914f-d7574a1deef5","Type":"ContainerDied","Data":"8daa1a07168f774b19f950db44ae3bd571b85182d5607f79915383f694c1a20d"} Oct 30 07:00:10.360443 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:10.360412 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" podUID="fc8e33ed-9831-4c77-8fdd-b64f9a593b99" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:10.652757 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:10.652686 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:10.789950 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:10.789919 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:10.992940 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:10.992844 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:11.189931 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.189909 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" Oct 30 07:00:11.339481 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.339409 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/108a685d-1c86-402f-914f-d7574a1deef5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/108a685d-1c86-402f-914f-d7574a1deef5-maas-api-xks-ca\") pod \"108a685d-1c86-402f-914f-d7574a1deef5\" (UID: \"108a685d-1c86-402f-914f-d7574a1deef5\") " Oct 30 07:00:11.339481 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.339456 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108a685d-1c86-402f-914f-d7574a1deef5-kube-api-access-pr27b\" (UniqueName: \"kubernetes.io/projected/108a685d-1c86-402f-914f-d7574a1deef5-kube-api-access-pr27b\") pod \"108a685d-1c86-402f-914f-d7574a1deef5\" (UID: \"108a685d-1c86-402f-914f-d7574a1deef5\") " Oct 30 07:00:11.339718 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.339524 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108a685d-1c86-402f-914f-d7574a1deef5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/108a685d-1c86-402f-914f-d7574a1deef5-maas-api-openshift-ca\") pod \"108a685d-1c86-402f-914f-d7574a1deef5\" (UID: \"108a685d-1c86-402f-914f-d7574a1deef5\") " Oct 30 07:00:11.339921 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.339890 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108a685d-1c86-402f-914f-d7574a1deef5-maas-api-openshift-ca" pod "108a685d-1c86-402f-914f-d7574a1deef5" (UID: "108a685d-1c86-402f-914f-d7574a1deef5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:11.341578 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.341554 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/108a685d-1c86-402f-914f-d7574a1deef5-maas-api-xks-ca" pod "108a685d-1c86-402f-914f-d7574a1deef5" (UID: "108a685d-1c86-402f-914f-d7574a1deef5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:11.341837 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.341820 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108a685d-1c86-402f-914f-d7574a1deef5-kube-api-access-pr27b" pod "108a685d-1c86-402f-914f-d7574a1deef5" (UID: "108a685d-1c86-402f-914f-d7574a1deef5"). InnerVolumeSpecName "kube-api-access-pr27b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:11.440143 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.440116 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/108a685d-1c86-402f-914f-d7574a1deef5-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:11.440143 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.440141 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr27b\" (UniqueName: \"kubernetes.io/projected/108a685d-1c86-402f-914f-d7574a1deef5-kube-api-access-pr27b\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:11.440309 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.440152 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/108a685d-1c86-402f-914f-d7574a1deef5-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:11.492245 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.492200 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:11.732251 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.732223 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9"] Oct 30 07:00:11.732692 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.732679 6999 cpu_manager.go:414] "containerMap: removing container" podUID="108a685d-1c86-402f-914f-d7574a1deef5" containerName="revoke-keys" Oct 30 07:00:11.732737 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.732694 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108a685d-1c86-402f-914f-d7574a1deef5" containerName="revoke-keys" Oct 30 07:00:11.732784 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.732776 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="108a685d-1c86-402f-914f-d7574a1deef5" containerName="revoke-keys" Oct 30 07:00:11.736073 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.736046 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" Oct 30 07:00:11.756332 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.756306 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9"] Oct 30 07:00:11.843353 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.843322 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/256d51fb-6115-40df-aa29-35fcde48dfb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9\" (UID: \"256d51fb-6115-40df-aa29-35fcde48dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" Oct 30 07:00:11.843525 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.843450 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdtcz\" (UniqueName: \"kubernetes.io/projected/256d51fb-6115-40df-aa29-35fcde48dfb3-kube-api-access-vdtcz\") pod \"maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9\" (UID: \"256d51fb-6115-40df-aa29-35fcde48dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" Oct 30 07:00:11.843525 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.843509 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/256d51fb-6115-40df-aa29-35fcde48dfb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9\" (UID: \"256d51fb-6115-40df-aa29-35fcde48dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" Oct 30 07:00:11.944517 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.944447 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/256d51fb-6115-40df-aa29-35fcde48dfb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9\" (UID: \"256d51fb-6115-40df-aa29-35fcde48dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" Oct 30 07:00:11.944660 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.944539 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdtcz\" (UniqueName: \"kubernetes.io/projected/256d51fb-6115-40df-aa29-35fcde48dfb3-kube-api-access-vdtcz\") pod \"maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9\" (UID: \"256d51fb-6115-40df-aa29-35fcde48dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" Oct 30 07:00:11.944660 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.944579 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/256d51fb-6115-40df-aa29-35fcde48dfb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9\" (UID: \"256d51fb-6115-40df-aa29-35fcde48dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" Oct 30 07:00:11.945255 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.945236 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/256d51fb-6115-40df-aa29-35fcde48dfb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9\" (UID: \"256d51fb-6115-40df-aa29-35fcde48dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" Oct 30 07:00:11.946966 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.946942 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/256d51fb-6115-40df-aa29-35fcde48dfb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9\" (UID: \"256d51fb-6115-40df-aa29-35fcde48dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" Oct 30 07:00:11.970065 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:11.970040 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdtcz\" (UniqueName: \"kubernetes.io/projected/256d51fb-6115-40df-aa29-35fcde48dfb3-kube-api-access-vdtcz\") pod \"maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9\" (UID: \"256d51fb-6115-40df-aa29-35fcde48dfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" Oct 30 07:00:12.048820 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:12.048800 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" Oct 30 07:00:12.054124 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:12.054105 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" Oct 30 07:00:12.054236 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:12.054128 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj" event={"ID":"108a685d-1c86-402f-914f-d7574a1deef5","Type":"ContainerDied","Data":"4c00002f05fc55748fed9fdb8e411abfc756e08bcec2aa2c6996736047a4c46f"} Oct 30 07:00:12.054236 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:12.054176 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c00002f05fc55748fed9fdb8e411abfc756e08bcec2aa2c6996736047a4c46f" Oct 30 07:00:12.350174 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:12.350151 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9"] Oct 30 07:00:12.352625 ip-10-0-131-226 kubenswrapper[6999]: W1030 07:00:12.352590 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256d51fb_6115_40df_aa29_35fcde48dfb3.slice/crio-b5b6e4c2b4ff68eee6a3b3cd0766b7104bb17ebfe018c3ee04828929818108a8 WatchSource:0}: Error finding container b5b6e4c2b4ff68eee6a3b3cd0766b7104bb17ebfe018c3ee04828929818108a8: Status 404 returned error can't find the container with id b5b6e4c2b4ff68eee6a3b3cd0766b7104bb17ebfe018c3ee04828929818108a8 Oct 30 07:00:12.761112 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:12.761072 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:13.019673 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:13.019602 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb"] Oct 30 07:00:13.034375 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:13.034342 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx"] Oct 30 07:00:13.061086 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:13.061057 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" event={"ID":"256d51fb-6115-40df-aa29-35fcde48dfb3","Type":"ContainerStarted","Data":"3c021a90374e7db525dc0ac71edeaa8dc61475f3cfdf91051773840fdd9df575"} Oct 30 07:00:13.061271 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:13.061111 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" event={"ID":"256d51fb-6115-40df-aa29-35fcde48dfb3","Type":"ContainerStarted","Data":"b5b6e4c2b4ff68eee6a3b3cd0766b7104bb17ebfe018c3ee04828929818108a8"} Oct 30 07:00:13.112249 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:13.112187 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" podStartSLOduration=2.112174716 podStartE2EDuration="2.112174716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:13.110951988 +0000 UTC m=+3979.619971877" watchObservedRunningTime="2026-10-30 07:00:13.112174716 +0000 UTC m=+3979.621194603" Oct 30 07:00:13.166609 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:13.166568 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:14.068200 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:14.068168 6999 generic.go:356] "Generic (PLEG): container finished" podID="256d51fb-6115-40df-aa29-35fcde48dfb3" containerID="3c021a90374e7db525dc0ac71edeaa8dc61475f3cfdf91051773840fdd9df575" exitCode=0 Oct 30 07:00:14.068592 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:14.068248 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" event={"ID":"256d51fb-6115-40df-aa29-35fcde48dfb3","Type":"ContainerDied","Data":"3c021a90374e7db525dc0ac71edeaa8dc61475f3cfdf91051773840fdd9df575"} Oct 30 07:00:14.185329 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:14.185297 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq"] Oct 30 07:00:14.320333 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:14.320255 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:14.456426 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:14.456395 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:14.747900 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:14.747868 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:15.203697 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:15.203673 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" Oct 30 07:00:15.379060 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:15.379037 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/256d51fb-6115-40df-aa29-35fcde48dfb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/256d51fb-6115-40df-aa29-35fcde48dfb3-maas-api-xks-ca\") pod \"256d51fb-6115-40df-aa29-35fcde48dfb3\" (UID: \"256d51fb-6115-40df-aa29-35fcde48dfb3\") " Oct 30 07:00:15.379264 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:15.379072 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/256d51fb-6115-40df-aa29-35fcde48dfb3-kube-api-access-vdtcz\" (UniqueName: \"kubernetes.io/projected/256d51fb-6115-40df-aa29-35fcde48dfb3-kube-api-access-vdtcz\") pod \"256d51fb-6115-40df-aa29-35fcde48dfb3\" (UID: \"256d51fb-6115-40df-aa29-35fcde48dfb3\") " Oct 30 07:00:15.379264 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:15.379168 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/256d51fb-6115-40df-aa29-35fcde48dfb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/256d51fb-6115-40df-aa29-35fcde48dfb3-maas-api-openshift-ca\") pod \"256d51fb-6115-40df-aa29-35fcde48dfb3\" (UID: \"256d51fb-6115-40df-aa29-35fcde48dfb3\") " Oct 30 07:00:15.379579 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:15.379557 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/256d51fb-6115-40df-aa29-35fcde48dfb3-maas-api-openshift-ca" pod "256d51fb-6115-40df-aa29-35fcde48dfb3" (UID: "256d51fb-6115-40df-aa29-35fcde48dfb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:15.381203 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:15.381183 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/256d51fb-6115-40df-aa29-35fcde48dfb3-maas-api-xks-ca" pod "256d51fb-6115-40df-aa29-35fcde48dfb3" (UID: "256d51fb-6115-40df-aa29-35fcde48dfb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:15.381410 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:15.381395 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/256d51fb-6115-40df-aa29-35fcde48dfb3-kube-api-access-vdtcz" pod "256d51fb-6115-40df-aa29-35fcde48dfb3" (UID: "256d51fb-6115-40df-aa29-35fcde48dfb3"). InnerVolumeSpecName "kube-api-access-vdtcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:15.479979 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:15.479955 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/256d51fb-6115-40df-aa29-35fcde48dfb3-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:15.479979 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:15.479982 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdtcz\" (UniqueName: \"kubernetes.io/projected/256d51fb-6115-40df-aa29-35fcde48dfb3-kube-api-access-vdtcz\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:15.480169 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:15.479994 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/256d51fb-6115-40df-aa29-35fcde48dfb3-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:15.601865 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:15.601836 6999 scope.go:122] "RemoveContainer" containerID="c535fccbd8379ea94468d87c1c221d041dab588e529ab16d8f446620067b29bf" Oct 30 07:00:15.602044 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:15.602026 6999 scope.go:122] "RemoveContainer" containerID="c2173f60acd0c4f9573084edee454f1f9029b7adbcbeafceea82a24ef9210578" Oct 30 07:00:16.080952 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:16.080921 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" event={"ID":"b5b8d268-3b87-4153-9759-a902e5dd3151","Type":"ContainerStarted","Data":"845ee98a2995cde96c54448426ae2c72ddc77d814fc97d247fdac953ad1aca0d"} Oct 30 07:00:16.082798 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:16.082773 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" event={"ID":"f467d292-2ba6-4287-be24-9016afdc01b0","Type":"ContainerStarted","Data":"c71a37b8ee34941cee4ab77f3afdf4213639fef330b2fae823dc4e430f29966a"} Oct 30 07:00:16.084127 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:16.084104 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" event={"ID":"256d51fb-6115-40df-aa29-35fcde48dfb3","Type":"ContainerDied","Data":"b5b6e4c2b4ff68eee6a3b3cd0766b7104bb17ebfe018c3ee04828929818108a8"} Oct 30 07:00:16.084127 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:16.084117 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9" Oct 30 07:00:16.084311 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:16.084143 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5b6e4c2b4ff68eee6a3b3cd0766b7104bb17ebfe018c3ee04828929818108a8" Oct 30 07:00:16.124958 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:16.124902 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" podStartSLOduration=16.124886006 podStartE2EDuration="16.124886006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:16.122589662 +0000 UTC m=+3982.631609550" watchObservedRunningTime="2026-10-30 07:00:16.124886006 +0000 UTC m=+3982.633905894" Oct 30 07:00:16.180050 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:16.179992 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" podStartSLOduration=16.179977616 podStartE2EDuration="16.179977616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:16.179839234 +0000 UTC m=+3982.688859122" watchObservedRunningTime="2026-10-30 07:00:16.179977616 +0000 UTC m=+3982.688997505" Oct 30 07:00:16.440416 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:16.440390 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht"] Oct 30 07:00:16.546727 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:16.546699 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:16.686624 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:16.686546 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:17.091852 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:17.091774 6999 generic.go:356] "Generic (PLEG): container finished" podID="b5b8d268-3b87-4153-9759-a902e5dd3151" containerID="845ee98a2995cde96c54448426ae2c72ddc77d814fc97d247fdac953ad1aca0d" exitCode=60 Oct 30 07:00:17.092022 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:17.091843 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" event={"ID":"b5b8d268-3b87-4153-9759-a902e5dd3151","Type":"ContainerDied","Data":"845ee98a2995cde96c54448426ae2c72ddc77d814fc97d247fdac953ad1aca0d"} Oct 30 07:00:17.092022 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:17.091900 6999 scope.go:122] "RemoveContainer" containerID="c2173f60acd0c4f9573084edee454f1f9029b7adbcbeafceea82a24ef9210578" Oct 30 07:00:17.092287 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:17.092269 6999 scope.go:122] "RemoveContainer" containerID="845ee98a2995cde96c54448426ae2c72ddc77d814fc97d247fdac953ad1aca0d" Oct 30 07:00:17.092550 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:00:17.092532 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889060-fffzs_odh-ai-gateway-infra(b5b8d268-3b87-4153-9759-a902e5dd3151)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" podUID="b5b8d268-3b87-4153-9759-a902e5dd3151" Oct 30 07:00:17.094101 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:17.094076 6999 generic.go:356] "Generic (PLEG): container finished" podID="f467d292-2ba6-4287-be24-9016afdc01b0" containerID="c71a37b8ee34941cee4ab77f3afdf4213639fef330b2fae823dc4e430f29966a" exitCode=60 Oct 30 07:00:17.094202 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:17.094138 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" event={"ID":"f467d292-2ba6-4287-be24-9016afdc01b0","Type":"ContainerDied","Data":"c71a37b8ee34941cee4ab77f3afdf4213639fef330b2fae823dc4e430f29966a"} Oct 30 07:00:17.094480 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:17.094466 6999 scope.go:122] "RemoveContainer" containerID="c71a37b8ee34941cee4ab77f3afdf4213639fef330b2fae823dc4e430f29966a" Oct 30 07:00:17.094697 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:00:17.094681 6999 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf_odh-ai-gateway-infra(f467d292-2ba6-4287-be24-9016afdc01b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" podUID="f467d292-2ba6-4287-be24-9016afdc01b0" Oct 30 07:00:17.104334 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:17.104232 6999 scope.go:122] "RemoveContainer" containerID="c535fccbd8379ea94468d87c1c221d041dab588e529ab16d8f446620067b29bf" Oct 30 07:00:17.134258 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:17.134231 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs"] Oct 30 07:00:17.190398 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:17.190368 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf"] Oct 30 07:00:18.065064 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.065032 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964"] Oct 30 07:00:18.065589 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.065251 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" podUID="bb3ede4f-5a78-4223-837f-b0186e9be925" containerName="maas-api" containerID="cri-o://5d9f6d7328557fceffc2000e474fe836033774ed1d5df3bf6a9de9bfa6c9b666" gracePeriod=30 Oct 30 07:00:18.168507 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.166768 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:18.288966 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.288718 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" Oct 30 07:00:18.354190 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.345886 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" Oct 30 07:00:18.409073 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.409042 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f467d292-2ba6-4287-be24-9016afdc01b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f467d292-2ba6-4287-be24-9016afdc01b0-maas-api-service-ca\") pod \"f467d292-2ba6-4287-be24-9016afdc01b0\" (UID: \"f467d292-2ba6-4287-be24-9016afdc01b0\") " Oct 30 07:00:18.409637 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.409614 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f467d292-2ba6-4287-be24-9016afdc01b0-kube-api-access-55jqk\" (UniqueName: \"kubernetes.io/projected/f467d292-2ba6-4287-be24-9016afdc01b0-kube-api-access-55jqk\") pod \"f467d292-2ba6-4287-be24-9016afdc01b0\" (UID: \"f467d292-2ba6-4287-be24-9016afdc01b0\") " Oct 30 07:00:18.410694 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.410670 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f467d292-2ba6-4287-be24-9016afdc01b0-maas-api-service-ca" pod "f467d292-2ba6-4287-be24-9016afdc01b0" (UID: "f467d292-2ba6-4287-be24-9016afdc01b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:18.410806 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.410786 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f467d292-2ba6-4287-be24-9016afdc01b0-maas-api-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:18.412905 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.412889 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f467d292-2ba6-4287-be24-9016afdc01b0-kube-api-access-55jqk" pod "f467d292-2ba6-4287-be24-9016afdc01b0" (UID: "f467d292-2ba6-4287-be24-9016afdc01b0"). InnerVolumeSpecName "kube-api-access-55jqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:18.419694 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.419667 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 07:00:18.511551 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.511519 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb3ede4f-5a78-4223-837f-b0186e9be925-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3ede4f-5a78-4223-837f-b0186e9be925-openshift-service-ca\") pod \"bb3ede4f-5a78-4223-837f-b0186e9be925\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " Oct 30 07:00:18.511747 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.511579 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb3ede4f-5a78-4223-837f-b0186e9be925-kube-api-access-smzm5\" (UniqueName: \"kubernetes.io/projected/bb3ede4f-5a78-4223-837f-b0186e9be925-kube-api-access-smzm5\") pod \"bb3ede4f-5a78-4223-837f-b0186e9be925\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " Oct 30 07:00:18.511747 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.511631 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb3ede4f-5a78-4223-837f-b0186e9be925-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3ede4f-5a78-4223-837f-b0186e9be925-maas-api-tls\") pod \"bb3ede4f-5a78-4223-837f-b0186e9be925\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " Oct 30 07:00:18.511747 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.511677 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5b8d268-3b87-4153-9759-a902e5dd3151-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b8d268-3b87-4153-9759-a902e5dd3151-maas-api-service-ca\") pod \"b5b8d268-3b87-4153-9759-a902e5dd3151\" (UID: \"b5b8d268-3b87-4153-9759-a902e5dd3151\") " Oct 30 07:00:18.511747 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.511717 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb3ede4f-5a78-4223-837f-b0186e9be925-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3ede4f-5a78-4223-837f-b0186e9be925-metrics-certs\") pod \"bb3ede4f-5a78-4223-837f-b0186e9be925\" (UID: \"bb3ede4f-5a78-4223-837f-b0186e9be925\") " Oct 30 07:00:18.512088 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.511763 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5b8d268-3b87-4153-9759-a902e5dd3151-kube-api-access-8qhcm\" (UniqueName: \"kubernetes.io/projected/b5b8d268-3b87-4153-9759-a902e5dd3151-kube-api-access-8qhcm\") pod \"b5b8d268-3b87-4153-9759-a902e5dd3151\" (UID: \"b5b8d268-3b87-4153-9759-a902e5dd3151\") " Oct 30 07:00:18.512088 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.511984 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3ede4f-5a78-4223-837f-b0186e9be925-openshift-service-ca" pod "bb3ede4f-5a78-4223-837f-b0186e9be925" (UID: "bb3ede4f-5a78-4223-837f-b0186e9be925"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:18.512206 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.512157 6999 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3ede4f-5a78-4223-837f-b0186e9be925-openshift-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:18.512206 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.512179 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55jqk\" (UniqueName: \"kubernetes.io/projected/f467d292-2ba6-4287-be24-9016afdc01b0-kube-api-access-55jqk\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:18.512688 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.512664 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b8d268-3b87-4153-9759-a902e5dd3151-maas-api-service-ca" pod "b5b8d268-3b87-4153-9759-a902e5dd3151" (UID: "b5b8d268-3b87-4153-9759-a902e5dd3151"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:18.514657 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.514599 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3ede4f-5a78-4223-837f-b0186e9be925-maas-api-tls" pod "bb3ede4f-5a78-4223-837f-b0186e9be925" (UID: "bb3ede4f-5a78-4223-837f-b0186e9be925"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:18.514761 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.514700 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b8d268-3b87-4153-9759-a902e5dd3151-kube-api-access-8qhcm" pod "b5b8d268-3b87-4153-9759-a902e5dd3151" (UID: "b5b8d268-3b87-4153-9759-a902e5dd3151"). InnerVolumeSpecName "kube-api-access-8qhcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:18.514888 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.514869 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb3ede4f-5a78-4223-837f-b0186e9be925-metrics-certs" pod "bb3ede4f-5a78-4223-837f-b0186e9be925" (UID: "bb3ede4f-5a78-4223-837f-b0186e9be925"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:18.515009 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.514988 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb3ede4f-5a78-4223-837f-b0186e9be925-kube-api-access-smzm5" pod "bb3ede4f-5a78-4223-837f-b0186e9be925" (UID: "bb3ede4f-5a78-4223-837f-b0186e9be925"). InnerVolumeSpecName "kube-api-access-smzm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:18.613195 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.613171 6999 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3ede4f-5a78-4223-837f-b0186e9be925-metrics-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:18.613195 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.613197 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qhcm\" (UniqueName: \"kubernetes.io/projected/b5b8d268-3b87-4153-9759-a902e5dd3151-kube-api-access-8qhcm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:18.613418 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.613207 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smzm5\" (UniqueName: \"kubernetes.io/projected/bb3ede4f-5a78-4223-837f-b0186e9be925-kube-api-access-smzm5\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:18.613418 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.613235 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3ede4f-5a78-4223-837f-b0186e9be925-maas-api-tls\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:18.613418 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.613249 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5b8d268-3b87-4153-9759-a902e5dd3151-maas-api-service-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:18.759672 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.759639 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:18.869549 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.869473 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl"] Oct 30 07:00:18.872558 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.872534 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fa5a69-pdgrl"] Oct 30 07:00:18.904363 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:18.904326 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:19.049143 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.049112 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:19.111839 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.111806 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" Oct 30 07:00:19.112310 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.111841 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf" event={"ID":"f467d292-2ba6-4287-be24-9016afdc01b0","Type":"ContainerDied","Data":"10b83d8dbd1bea7f1848affdfd4745226f40df80f972241bcad222cc19897406"} Oct 30 07:00:19.112310 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.111900 6999 scope.go:122] "RemoveContainer" containerID="c71a37b8ee34941cee4ab77f3afdf4213639fef330b2fae823dc4e430f29966a" Oct 30 07:00:19.113576 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.113552 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" event={"ID":"b5b8d268-3b87-4153-9759-a902e5dd3151","Type":"ContainerDied","Data":"6e0793f14eefcd6a6aa6f243b55b18dc140b78962c03ac22cf7d613c3bb2ce64"} Oct 30 07:00:19.113661 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.113645 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs" Oct 30 07:00:19.121609 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.121592 6999 scope.go:122] "RemoveContainer" containerID="845ee98a2995cde96c54448426ae2c72ddc77d814fc97d247fdac953ad1aca0d" Oct 30 07:00:19.122818 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.122794 6999 generic.go:356] "Generic (PLEG): container finished" podID="bb3ede4f-5a78-4223-837f-b0186e9be925" containerID="5d9f6d7328557fceffc2000e474fe836033774ed1d5df3bf6a9de9bfa6c9b666" exitCode=0 Oct 30 07:00:19.122903 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.122825 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" event={"ID":"bb3ede4f-5a78-4223-837f-b0186e9be925","Type":"ContainerDied","Data":"5d9f6d7328557fceffc2000e474fe836033774ed1d5df3bf6a9de9bfa6c9b666"} Oct 30 07:00:19.122903 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.122849 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" Oct 30 07:00:19.122903 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.122872 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964" event={"ID":"bb3ede4f-5a78-4223-837f-b0186e9be925","Type":"ContainerDied","Data":"44f12ea9fa26d6e031a553919ee9e1b1d75e9620ab950b34b6020065fe600fad"} Oct 30 07:00:19.134398 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.134329 6999 scope.go:122] "RemoveContainer" containerID="5d9f6d7328557fceffc2000e474fe836033774ed1d5df3bf6a9de9bfa6c9b666" Oct 30 07:00:19.143381 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.143361 6999 scope.go:122] "RemoveContainer" containerID="5d9f6d7328557fceffc2000e474fe836033774ed1d5df3bf6a9de9bfa6c9b666" Oct 30 07:00:19.143634 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:00:19.143616 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9f6d7328557fceffc2000e474fe836033774ed1d5df3bf6a9de9bfa6c9b666\": container with ID starting with 5d9f6d7328557fceffc2000e474fe836033774ed1d5df3bf6a9de9bfa6c9b666 not found: ID does not exist" containerID="5d9f6d7328557fceffc2000e474fe836033774ed1d5df3bf6a9de9bfa6c9b666" Oct 30 07:00:19.143706 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.143645 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9f6d7328557fceffc2000e474fe836033774ed1d5df3bf6a9de9bfa6c9b666"} err="failed to get container status \"5d9f6d7328557fceffc2000e474fe836033774ed1d5df3bf6a9de9bfa6c9b666\": rpc error: code = NotFound desc = could not find container \"5d9f6d7328557fceffc2000e474fe836033774ed1d5df3bf6a9de9bfa6c9b666\": container with ID starting with 5d9f6d7328557fceffc2000e474fe836033774ed1d5df3bf6a9de9bfa6c9b666 not found: ID does not exist" Oct 30 07:00:19.161792 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.161771 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs"] Oct 30 07:00:19.166593 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.166572 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889060-fffzs"] Oct 30 07:00:19.185781 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.185747 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" podUID="f977329f-2b08-42f3-9359-154994886be9" containerName="payload-pre-processing" containerID="cri-o://a9f73eeadaa81e4433aa93402a85a2648d6f9e3fbb882eeb7b22cbfcb8ad85ab" gracePeriod=25 Oct 30 07:00:19.216809 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.216780 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf"] Oct 30 07:00:19.225768 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.225742 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aaf467-29889060-9zpjf"] Oct 30 07:00:19.260912 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.260888 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964"] Oct 30 07:00:19.264643 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.264621 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-fa5a69-74f5c9456c-m8964"] Oct 30 07:00:19.429045 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.428958 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:19.608385 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.608351 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11cdb156-da38-4a6a-8976-f0a1a2eff985" path="/var/lib/kubelet/pods/11cdb156-da38-4a6a-8976-f0a1a2eff985/volumes" Oct 30 07:00:19.608797 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.608781 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5b8d268-3b87-4153-9759-a902e5dd3151" path="/var/lib/kubelet/pods/b5b8d268-3b87-4153-9759-a902e5dd3151/volumes" Oct 30 07:00:19.609235 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.609204 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb3ede4f-5a78-4223-837f-b0186e9be925" path="/var/lib/kubelet/pods/bb3ede4f-5a78-4223-837f-b0186e9be925/volumes" Oct 30 07:00:19.609928 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:19.609913 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f467d292-2ba6-4287-be24-9016afdc01b0" path="/var/lib/kubelet/pods/f467d292-2ba6-4287-be24-9016afdc01b0/volumes" Oct 30 07:00:20.360453 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:20.360417 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" podUID="fc8e33ed-9831-4c77-8fdd-b64f9a593b99" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:20.970082 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:20.970050 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:22.397205 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:22.397174 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:22.528767 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:22.528730 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:22.740030 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:22.739948 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:23.121310 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.121280 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" Oct 30 07:00:23.149900 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.149871 6999 generic.go:356] "Generic (PLEG): container finished" podID="fc8e33ed-9831-4c77-8fdd-b64f9a593b99" containerID="88d16e753a568ded154fb0bce97634a071c074a264845c5f6033c3220bc92eb4" exitCode=0 Oct 30 07:00:23.150083 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.149970 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" Oct 30 07:00:23.150083 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.150014 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" event={"ID":"fc8e33ed-9831-4c77-8fdd-b64f9a593b99","Type":"ContainerDied","Data":"88d16e753a568ded154fb0bce97634a071c074a264845c5f6033c3220bc92eb4"} Oct 30 07:00:23.150083 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.150074 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7" event={"ID":"fc8e33ed-9831-4c77-8fdd-b64f9a593b99","Type":"ContainerDied","Data":"59a0a1c8dc65864e9ca45ab361eb423ff16ddeaaee54e49d4a95b2626d79d8b9"} Oct 30 07:00:23.150283 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.150106 6999 scope.go:122] "RemoveContainer" containerID="88d16e753a568ded154fb0bce97634a071c074a264845c5f6033c3220bc92eb4" Oct 30 07:00:23.159458 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.159439 6999 scope.go:122] "RemoveContainer" containerID="88d16e753a568ded154fb0bce97634a071c074a264845c5f6033c3220bc92eb4" Oct 30 07:00:23.159754 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:00:23.159728 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d16e753a568ded154fb0bce97634a071c074a264845c5f6033c3220bc92eb4\": container with ID starting with 88d16e753a568ded154fb0bce97634a071c074a264845c5f6033c3220bc92eb4 not found: ID does not exist" containerID="88d16e753a568ded154fb0bce97634a071c074a264845c5f6033c3220bc92eb4" Oct 30 07:00:23.159822 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.159765 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d16e753a568ded154fb0bce97634a071c074a264845c5f6033c3220bc92eb4"} err="failed to get container status \"88d16e753a568ded154fb0bce97634a071c074a264845c5f6033c3220bc92eb4\": rpc error: code = NotFound desc = could not find container \"88d16e753a568ded154fb0bce97634a071c074a264845c5f6033c3220bc92eb4\": container with ID starting with 88d16e753a568ded154fb0bce97634a071c074a264845c5f6033c3220bc92eb4 not found: ID does not exist" Oct 30 07:00:23.166201 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.166179 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc8e33ed-9831-4c77-8fdd-b64f9a593b99-kube-api-access-6r77k\" (UniqueName: \"kubernetes.io/projected/fc8e33ed-9831-4c77-8fdd-b64f9a593b99-kube-api-access-6r77k\") pod \"fc8e33ed-9831-4c77-8fdd-b64f9a593b99\" (UID: \"fc8e33ed-9831-4c77-8fdd-b64f9a593b99\") " Oct 30 07:00:23.166404 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.166387 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc8e33ed-9831-4c77-8fdd-b64f9a593b99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8e33ed-9831-4c77-8fdd-b64f9a593b99-plugins-config-volume\") pod \"fc8e33ed-9831-4c77-8fdd-b64f9a593b99\" (UID: \"fc8e33ed-9831-4c77-8fdd-b64f9a593b99\") " Oct 30 07:00:23.167047 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.167025 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8e33ed-9831-4c77-8fdd-b64f9a593b99-plugins-config-volume" pod "fc8e33ed-9831-4c77-8fdd-b64f9a593b99" (UID: "fc8e33ed-9831-4c77-8fdd-b64f9a593b99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:23.169653 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.169461 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8e33ed-9831-4c77-8fdd-b64f9a593b99-kube-api-access-6r77k" pod "fc8e33ed-9831-4c77-8fdd-b64f9a593b99" (UID: "fc8e33ed-9831-4c77-8fdd-b64f9a593b99"). InnerVolumeSpecName "kube-api-access-6r77k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:23.267646 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.267609 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc8e33ed-9831-4c77-8fdd-b64f9a593b99-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:23.267646 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.267637 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r77k\" (UniqueName: \"kubernetes.io/projected/fc8e33ed-9831-4c77-8fdd-b64f9a593b99-kube-api-access-6r77k\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:23.494917 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.494886 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7"] Oct 30 07:00:23.505876 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.505849 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-868b1f-6856b9dbff-xj5b7"] Oct 30 07:00:23.607536 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:23.607506 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc8e33ed-9831-4c77-8fdd-b64f9a593b99" path="/var/lib/kubelet/pods/fc8e33ed-9831-4c77-8fdd-b64f9a593b99/volumes" Oct 30 07:00:24.242970 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.242944 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq"] Oct 30 07:00:24.243382 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243366 6999 cpu_manager.go:414] "containerMap: removing container" podUID="256d51fb-6115-40df-aa29-35fcde48dfb3" containerName="revoke-keys" Oct 30 07:00:24.243382 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243381 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="256d51fb-6115-40df-aa29-35fcde48dfb3" containerName="revoke-keys" Oct 30 07:00:24.243481 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243402 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b5b8d268-3b87-4153-9759-a902e5dd3151" containerName="cleanup" Oct 30 07:00:24.243481 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243408 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b8d268-3b87-4153-9759-a902e5dd3151" containerName="cleanup" Oct 30 07:00:24.243481 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243417 6999 cpu_manager.go:414] "containerMap: removing container" podUID="f467d292-2ba6-4287-be24-9016afdc01b0" containerName="cleanup" Oct 30 07:00:24.243481 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243421 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f467d292-2ba6-4287-be24-9016afdc01b0" containerName="cleanup" Oct 30 07:00:24.243481 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243430 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b5b8d268-3b87-4153-9759-a902e5dd3151" containerName="cleanup" Oct 30 07:00:24.243481 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243436 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b8d268-3b87-4153-9759-a902e5dd3151" containerName="cleanup" Oct 30 07:00:24.243481 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243442 6999 cpu_manager.go:414] "containerMap: removing container" podUID="f467d292-2ba6-4287-be24-9016afdc01b0" containerName="cleanup" Oct 30 07:00:24.243481 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243447 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f467d292-2ba6-4287-be24-9016afdc01b0" containerName="cleanup" Oct 30 07:00:24.243481 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243457 6999 cpu_manager.go:414] "containerMap: removing container" podUID="fc8e33ed-9831-4c77-8fdd-b64f9a593b99" containerName="payload-pre-processing" Oct 30 07:00:24.243481 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243461 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc8e33ed-9831-4c77-8fdd-b64f9a593b99" containerName="payload-pre-processing" Oct 30 07:00:24.243481 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243476 6999 cpu_manager.go:414] "containerMap: removing container" podUID="b5b8d268-3b87-4153-9759-a902e5dd3151" containerName="cleanup" Oct 30 07:00:24.243481 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243480 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5b8d268-3b87-4153-9759-a902e5dd3151" containerName="cleanup" Oct 30 07:00:24.243883 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243490 6999 cpu_manager.go:414] "containerMap: removing container" podUID="bb3ede4f-5a78-4223-837f-b0186e9be925" containerName="maas-api" Oct 30 07:00:24.243883 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243495 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb3ede4f-5a78-4223-837f-b0186e9be925" containerName="maas-api" Oct 30 07:00:24.243883 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243558 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="256d51fb-6115-40df-aa29-35fcde48dfb3" containerName="revoke-keys" Oct 30 07:00:24.243883 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243569 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b8d268-3b87-4153-9759-a902e5dd3151" containerName="cleanup" Oct 30 07:00:24.243883 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243574 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="f467d292-2ba6-4287-be24-9016afdc01b0" containerName="cleanup" Oct 30 07:00:24.243883 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243579 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b8d268-3b87-4153-9759-a902e5dd3151" containerName="cleanup" Oct 30 07:00:24.243883 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243585 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="f467d292-2ba6-4287-be24-9016afdc01b0" containerName="cleanup" Oct 30 07:00:24.243883 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243591 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc8e33ed-9831-4c77-8fdd-b64f9a593b99" containerName="payload-pre-processing" Oct 30 07:00:24.243883 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243599 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5b8d268-3b87-4153-9759-a902e5dd3151" containerName="cleanup" Oct 30 07:00:24.243883 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.243604 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb3ede4f-5a78-4223-837f-b0186e9be925" containerName="maas-api" Oct 30 07:00:24.248032 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.248003 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" Oct 30 07:00:24.266773 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.266751 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:00:24.268917 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.268897 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq"] Oct 30 07:00:24.278080 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.278057 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbvwt\" (UniqueName: \"kubernetes.io/projected/5acb488c-537b-4fdc-9555-246b327de37d-kube-api-access-kbvwt\") pod \"maas-api-revoke-sub-d589320d-qhkmq\" (UID: \"5acb488c-537b-4fdc-9555-246b327de37d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" Oct 30 07:00:24.278258 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.278191 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5acb488c-537b-4fdc-9555-246b327de37d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d589320d-qhkmq\" (UID: \"5acb488c-537b-4fdc-9555-246b327de37d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" Oct 30 07:00:24.278329 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.278290 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5acb488c-537b-4fdc-9555-246b327de37d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d589320d-qhkmq\" (UID: \"5acb488c-537b-4fdc-9555-246b327de37d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" Oct 30 07:00:24.278853 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.278837 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 07:00:24.379638 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.379606 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5acb488c-537b-4fdc-9555-246b327de37d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d589320d-qhkmq\" (UID: \"5acb488c-537b-4fdc-9555-246b327de37d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" Oct 30 07:00:24.379845 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.379680 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5acb488c-537b-4fdc-9555-246b327de37d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d589320d-qhkmq\" (UID: \"5acb488c-537b-4fdc-9555-246b327de37d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" Oct 30 07:00:24.379845 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.379728 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbvwt\" (UniqueName: \"kubernetes.io/projected/5acb488c-537b-4fdc-9555-246b327de37d-kube-api-access-kbvwt\") pod \"maas-api-revoke-sub-d589320d-qhkmq\" (UID: \"5acb488c-537b-4fdc-9555-246b327de37d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" Oct 30 07:00:24.380270 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.380244 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5acb488c-537b-4fdc-9555-246b327de37d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d589320d-qhkmq\" (UID: \"5acb488c-537b-4fdc-9555-246b327de37d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" Oct 30 07:00:24.382161 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.382132 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5acb488c-537b-4fdc-9555-246b327de37d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d589320d-qhkmq\" (UID: \"5acb488c-537b-4fdc-9555-246b327de37d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" Oct 30 07:00:24.402740 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.402713 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbvwt\" (UniqueName: \"kubernetes.io/projected/5acb488c-537b-4fdc-9555-246b327de37d-kube-api-access-kbvwt\") pod \"maas-api-revoke-sub-d589320d-qhkmq\" (UID: \"5acb488c-537b-4fdc-9555-246b327de37d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" Oct 30 07:00:24.561470 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.561391 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" Oct 30 07:00:24.650077 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.650047 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:24.867668 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:24.867635 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq"] Oct 30 07:00:24.870864 ip-10-0-131-226 kubenswrapper[6999]: W1030 07:00:24.870832 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5acb488c_537b_4fdc_9555_246b327de37d.slice/crio-676ac3e3f7fc7ff2b21c5b6b8b95faa58d5968d6dc8d29df4b0e3f8f2abd2e4f WatchSource:0}: Error finding container 676ac3e3f7fc7ff2b21c5b6b8b95faa58d5968d6dc8d29df4b0e3f8f2abd2e4f: Status 404 returned error can't find the container with id 676ac3e3f7fc7ff2b21c5b6b8b95faa58d5968d6dc8d29df4b0e3f8f2abd2e4f Oct 30 07:00:25.163107 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:25.163076 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" event={"ID":"5acb488c-537b-4fdc-9555-246b327de37d","Type":"ContainerStarted","Data":"22d81cbd71723e35172e803df209dc14da0ae3190e1ea0b4901770bd282c2538"} Oct 30 07:00:25.163260 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:25.163126 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" event={"ID":"5acb488c-537b-4fdc-9555-246b327de37d","Type":"ContainerStarted","Data":"676ac3e3f7fc7ff2b21c5b6b8b95faa58d5968d6dc8d29df4b0e3f8f2abd2e4f"} Oct 30 07:00:25.204379 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:25.204336 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" podStartSLOduration=1.204322202 podStartE2EDuration="1.204322202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:25.20266651 +0000 UTC m=+3991.711686398" watchObservedRunningTime="2026-10-30 07:00:25.204322202 +0000 UTC m=+3991.713342125" Oct 30 07:00:25.890834 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:25.890792 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:26.024452 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.024371 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:26.095302 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.095269 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs"] Oct 30 07:00:26.095806 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.095786 6999 cpu_manager.go:414] "containerMap: removing container" podUID="f467d292-2ba6-4287-be24-9016afdc01b0" containerName="cleanup" Oct 30 07:00:26.095906 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.095808 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f467d292-2ba6-4287-be24-9016afdc01b0" containerName="cleanup" Oct 30 07:00:26.095973 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.095956 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="f467d292-2ba6-4287-be24-9016afdc01b0" containerName="cleanup" Oct 30 07:00:26.099336 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.099313 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" Oct 30 07:00:26.123645 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.123619 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs"] Oct 30 07:00:26.176764 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.176727 6999 generic.go:356] "Generic (PLEG): container finished" podID="5acb488c-537b-4fdc-9555-246b327de37d" containerID="22d81cbd71723e35172e803df209dc14da0ae3190e1ea0b4901770bd282c2538" exitCode=0 Oct 30 07:00:26.177162 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.177079 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" event={"ID":"5acb488c-537b-4fdc-9555-246b327de37d","Type":"ContainerDied","Data":"22d81cbd71723e35172e803df209dc14da0ae3190e1ea0b4901770bd282c2538"} Oct 30 07:00:26.196376 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.196336 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d03989d-f895-4557-b293-4ae0313c0cc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e97edfa7-7jjcs\" (UID: \"7d03989d-f895-4557-b293-4ae0313c0cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" Oct 30 07:00:26.196516 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.196445 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d03989d-f895-4557-b293-4ae0313c0cc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e97edfa7-7jjcs\" (UID: \"7d03989d-f895-4557-b293-4ae0313c0cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" Oct 30 07:00:26.196563 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.196516 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx7rm\" (UniqueName: \"kubernetes.io/projected/7d03989d-f895-4557-b293-4ae0313c0cc2-kube-api-access-lx7rm\") pod \"maas-api-revoke-sub-e97edfa7-7jjcs\" (UID: \"7d03989d-f895-4557-b293-4ae0313c0cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" Oct 30 07:00:26.252424 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.252390 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:26.298288 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.298157 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d03989d-f895-4557-b293-4ae0313c0cc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e97edfa7-7jjcs\" (UID: \"7d03989d-f895-4557-b293-4ae0313c0cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" Oct 30 07:00:26.298288 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.298257 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d03989d-f895-4557-b293-4ae0313c0cc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e97edfa7-7jjcs\" (UID: \"7d03989d-f895-4557-b293-4ae0313c0cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" Oct 30 07:00:26.298505 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.298298 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx7rm\" (UniqueName: \"kubernetes.io/projected/7d03989d-f895-4557-b293-4ae0313c0cc2-kube-api-access-lx7rm\") pod \"maas-api-revoke-sub-e97edfa7-7jjcs\" (UID: \"7d03989d-f895-4557-b293-4ae0313c0cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" Oct 30 07:00:26.298981 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.298959 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d03989d-f895-4557-b293-4ae0313c0cc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e97edfa7-7jjcs\" (UID: \"7d03989d-f895-4557-b293-4ae0313c0cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" Oct 30 07:00:26.300752 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.300735 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d03989d-f895-4557-b293-4ae0313c0cc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e97edfa7-7jjcs\" (UID: \"7d03989d-f895-4557-b293-4ae0313c0cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" Oct 30 07:00:26.321605 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.321583 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx7rm\" (UniqueName: \"kubernetes.io/projected/7d03989d-f895-4557-b293-4ae0313c0cc2-kube-api-access-lx7rm\") pod \"maas-api-revoke-sub-e97edfa7-7jjcs\" (UID: \"7d03989d-f895-4557-b293-4ae0313c0cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" Oct 30 07:00:26.413350 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.413324 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" Oct 30 07:00:26.621356 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.621321 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:26.717376 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.717269 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs"] Oct 30 07:00:26.720177 ip-10-0-131-226 kubenswrapper[6999]: W1030 07:00:26.720150 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d03989d_f895_4557_b293_4ae0313c0cc2.slice/crio-f4c36798093a37320b655209b873e5d92b961aae2725918fe592372e4e7b2a47 WatchSource:0}: Error finding container f4c36798093a37320b655209b873e5d92b961aae2725918fe592372e4e7b2a47: Status 404 returned error can't find the container with id f4c36798093a37320b655209b873e5d92b961aae2725918fe592372e4e7b2a47 Oct 30 07:00:26.762612 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:26.762588 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:27.001160 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.001125 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g"] Oct 30 07:00:27.005196 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.005177 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" Oct 30 07:00:27.025986 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.025958 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g"] Oct 30 07:00:27.106319 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.106295 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvp6n\" (UniqueName: \"kubernetes.io/projected/6e53ae74-7c02-4cec-90e6-c5db61897c64-kube-api-access-dvp6n\") pod \"maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g\" (UID: \"6e53ae74-7c02-4cec-90e6-c5db61897c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" Oct 30 07:00:27.106439 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.106330 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e53ae74-7c02-4cec-90e6-c5db61897c64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g\" (UID: \"6e53ae74-7c02-4cec-90e6-c5db61897c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" Oct 30 07:00:27.106482 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.106434 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e53ae74-7c02-4cec-90e6-c5db61897c64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g\" (UID: \"6e53ae74-7c02-4cec-90e6-c5db61897c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" Oct 30 07:00:27.184597 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.184570 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" event={"ID":"7d03989d-f895-4557-b293-4ae0313c0cc2","Type":"ContainerStarted","Data":"36cb9ad642a079c9c6cafefa5bb5879377c81beb9f74e6b9e0b3cbd1f71d08e0"} Oct 30 07:00:27.184737 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.184631 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" event={"ID":"7d03989d-f895-4557-b293-4ae0313c0cc2","Type":"ContainerStarted","Data":"f4c36798093a37320b655209b873e5d92b961aae2725918fe592372e4e7b2a47"} Oct 30 07:00:27.207411 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.207349 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e53ae74-7c02-4cec-90e6-c5db61897c64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g\" (UID: \"6e53ae74-7c02-4cec-90e6-c5db61897c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" Oct 30 07:00:27.207536 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.207459 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvp6n\" (UniqueName: \"kubernetes.io/projected/6e53ae74-7c02-4cec-90e6-c5db61897c64-kube-api-access-dvp6n\") pod \"maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g\" (UID: \"6e53ae74-7c02-4cec-90e6-c5db61897c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" Oct 30 07:00:27.207536 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.207482 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e53ae74-7c02-4cec-90e6-c5db61897c64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g\" (UID: \"6e53ae74-7c02-4cec-90e6-c5db61897c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" Oct 30 07:00:27.208159 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.208139 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e53ae74-7c02-4cec-90e6-c5db61897c64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g\" (UID: \"6e53ae74-7c02-4cec-90e6-c5db61897c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" Oct 30 07:00:27.209799 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.209780 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e53ae74-7c02-4cec-90e6-c5db61897c64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g\" (UID: \"6e53ae74-7c02-4cec-90e6-c5db61897c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" Oct 30 07:00:27.226075 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.226022 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" podStartSLOduration=1.226004953 podStartE2EDuration="1.226004953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:27.224339696 +0000 UTC m=+3993.733359584" watchObservedRunningTime="2026-10-30 07:00:27.226004953 +0000 UTC m=+3993.735024841" Oct 30 07:00:27.232688 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.232662 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvp6n\" (UniqueName: \"kubernetes.io/projected/6e53ae74-7c02-4cec-90e6-c5db61897c64-kube-api-access-dvp6n\") pod \"maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g\" (UID: \"6e53ae74-7c02-4cec-90e6-c5db61897c64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" Oct 30 07:00:27.312133 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.312109 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" Oct 30 07:00:27.317929 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.317908 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" Oct 30 07:00:27.409461 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.409427 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5acb488c-537b-4fdc-9555-246b327de37d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5acb488c-537b-4fdc-9555-246b327de37d-maas-api-openshift-ca\") pod \"5acb488c-537b-4fdc-9555-246b327de37d\" (UID: \"5acb488c-537b-4fdc-9555-246b327de37d\") " Oct 30 07:00:27.409636 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.409513 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5acb488c-537b-4fdc-9555-246b327de37d-kube-api-access-kbvwt\" (UniqueName: \"kubernetes.io/projected/5acb488c-537b-4fdc-9555-246b327de37d-kube-api-access-kbvwt\") pod \"5acb488c-537b-4fdc-9555-246b327de37d\" (UID: \"5acb488c-537b-4fdc-9555-246b327de37d\") " Oct 30 07:00:27.409636 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.409547 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5acb488c-537b-4fdc-9555-246b327de37d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5acb488c-537b-4fdc-9555-246b327de37d-maas-api-xks-ca\") pod \"5acb488c-537b-4fdc-9555-246b327de37d\" (UID: \"5acb488c-537b-4fdc-9555-246b327de37d\") " Oct 30 07:00:27.409847 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.409820 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5acb488c-537b-4fdc-9555-246b327de37d-maas-api-openshift-ca" pod "5acb488c-537b-4fdc-9555-246b327de37d" (UID: "5acb488c-537b-4fdc-9555-246b327de37d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:27.411687 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.411662 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5acb488c-537b-4fdc-9555-246b327de37d-kube-api-access-kbvwt" pod "5acb488c-537b-4fdc-9555-246b327de37d" (UID: "5acb488c-537b-4fdc-9555-246b327de37d"). InnerVolumeSpecName "kube-api-access-kbvwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:27.411819 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.411707 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5acb488c-537b-4fdc-9555-246b327de37d-maas-api-xks-ca" pod "5acb488c-537b-4fdc-9555-246b327de37d" (UID: "5acb488c-537b-4fdc-9555-246b327de37d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:27.510401 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.510347 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5acb488c-537b-4fdc-9555-246b327de37d-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:27.510401 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.510370 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbvwt\" (UniqueName: \"kubernetes.io/projected/5acb488c-537b-4fdc-9555-246b327de37d-kube-api-access-kbvwt\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:27.510401 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.510380 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5acb488c-537b-4fdc-9555-246b327de37d-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:27.620682 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:27.620656 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g"] Oct 30 07:00:27.639078 ip-10-0-131-226 kubenswrapper[6999]: W1030 07:00:27.639053 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e53ae74_7c02_4cec_90e6_c5db61897c64.slice/crio-908fd8988b380cc9839ed5611c3aff4156242d8fb66c7a533d5feabe7a30d385 WatchSource:0}: Error finding container 908fd8988b380cc9839ed5611c3aff4156242d8fb66c7a533d5feabe7a30d385: Status 404 returned error can't find the container with id 908fd8988b380cc9839ed5611c3aff4156242d8fb66c7a533d5feabe7a30d385 Oct 30 07:00:28.024147 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.024113 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" podUID="f977329f-2b08-42f3-9359-154994886be9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:28.042788 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.042755 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" podUID="5990b367-6bd6-483b-ab76-35375b015cbd" containerName="main" containerID="cri-o://fe0eff16cc7e522dc908c92eecc7e21c73811dc06956cbddc6906e8009907acf" gracePeriod=45 Oct 30 07:00:28.047019 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.046994 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" podUID="5990b367-6bd6-483b-ab76-35375b015cbd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": EOF" Oct 30 07:00:28.055538 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.055513 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" podUID="8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" containerName="main" containerID="cri-o://405773f083aa8e5b74581b4f6ffd6f71a16028b93098ec9f1c3698b2786372dc" gracePeriod=45 Oct 30 07:00:28.137834 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.137804 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" podUID="8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused" Oct 30 07:00:28.191915 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.191880 6999 generic.go:356] "Generic (PLEG): container finished" podID="8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" containerID="405773f083aa8e5b74581b4f6ffd6f71a16028b93098ec9f1c3698b2786372dc" exitCode=0 Oct 30 07:00:28.192091 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.191959 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" event={"ID":"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3","Type":"ContainerDied","Data":"405773f083aa8e5b74581b4f6ffd6f71a16028b93098ec9f1c3698b2786372dc"} Oct 30 07:00:28.193707 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.193681 6999 generic.go:356] "Generic (PLEG): container finished" podID="5990b367-6bd6-483b-ab76-35375b015cbd" containerID="fe0eff16cc7e522dc908c92eecc7e21c73811dc06956cbddc6906e8009907acf" exitCode=0 Oct 30 07:00:28.193837 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.193819 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" event={"ID":"5990b367-6bd6-483b-ab76-35375b015cbd","Type":"ContainerDied","Data":"fe0eff16cc7e522dc908c92eecc7e21c73811dc06956cbddc6906e8009907acf"} Oct 30 07:00:28.195749 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.195605 6999 generic.go:356] "Generic (PLEG): container finished" podID="7d03989d-f895-4557-b293-4ae0313c0cc2" containerID="36cb9ad642a079c9c6cafefa5bb5879377c81beb9f74e6b9e0b3cbd1f71d08e0" exitCode=0 Oct 30 07:00:28.195749 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.195704 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" event={"ID":"7d03989d-f895-4557-b293-4ae0313c0cc2","Type":"ContainerDied","Data":"36cb9ad642a079c9c6cafefa5bb5879377c81beb9f74e6b9e0b3cbd1f71d08e0"} Oct 30 07:00:28.198569 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.197514 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" event={"ID":"6e53ae74-7c02-4cec-90e6-c5db61897c64","Type":"ContainerStarted","Data":"621ac90be0dae04d39af189d2dfab408baf7197b2849f80cec07b154c186353a"} Oct 30 07:00:28.198569 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.197562 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" event={"ID":"6e53ae74-7c02-4cec-90e6-c5db61897c64","Type":"ContainerStarted","Data":"908fd8988b380cc9839ed5611c3aff4156242d8fb66c7a533d5feabe7a30d385"} Oct 30 07:00:28.199954 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.199829 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" event={"ID":"5acb488c-537b-4fdc-9555-246b327de37d","Type":"ContainerDied","Data":"676ac3e3f7fc7ff2b21c5b6b8b95faa58d5968d6dc8d29df4b0e3f8f2abd2e4f"} Oct 30 07:00:28.199954 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.199874 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="676ac3e3f7fc7ff2b21c5b6b8b95faa58d5968d6dc8d29df4b0e3f8f2abd2e4f" Oct 30 07:00:28.199954 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.199931 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d589320d-qhkmq" Oct 30 07:00:28.301661 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.301635 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 07:00:28.310134 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.310113 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 07:00:28.314967 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.314920 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" podStartSLOduration=2.314905466 podStartE2EDuration="2.314905466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:28.314728756 +0000 UTC m=+3994.823748644" watchObservedRunningTime="2026-10-30 07:00:28.314905466 +0000 UTC m=+3994.823925356" Oct 30 07:00:28.418113 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.418036 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-tls-certs\") pod \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " Oct 30 07:00:28.418267 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.418144 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-dshm\") pod \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " Oct 30 07:00:28.418267 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.418193 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-model-cache\") pod \"5990b367-6bd6-483b-ab76-35375b015cbd\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " Oct 30 07:00:28.418267 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.418261 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5990b367-6bd6-483b-ab76-35375b015cbd-tls-certs\" (UniqueName: \"kubernetes.io/secret/5990b367-6bd6-483b-ab76-35375b015cbd-tls-certs\") pod \"5990b367-6bd6-483b-ab76-35375b015cbd\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " Oct 30 07:00:28.418462 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.418290 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-home\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-home\") pod \"5990b367-6bd6-483b-ab76-35375b015cbd\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " Oct 30 07:00:28.418462 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.418316 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-tmp-dir\") pod \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " Oct 30 07:00:28.418462 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.418342 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-kube-api-access-qwvkf\" (UniqueName: \"kubernetes.io/projected/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-kube-api-access-qwvkf\") pod \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " Oct 30 07:00:28.418462 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.418376 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-model-cache\") pod \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " Oct 30 07:00:28.418462 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.418417 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5990b367-6bd6-483b-ab76-35375b015cbd-kube-api-access-h9d8k\" (UniqueName: \"kubernetes.io/projected/5990b367-6bd6-483b-ab76-35375b015cbd-kube-api-access-h9d8k\") pod \"5990b367-6bd6-483b-ab76-35375b015cbd\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " Oct 30 07:00:28.418740 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.418464 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-home\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-home\") pod \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\" (UID: \"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3\") " Oct 30 07:00:28.418740 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.418492 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-tmp-dir\") pod \"5990b367-6bd6-483b-ab76-35375b015cbd\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " Oct 30 07:00:28.418740 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.418525 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-dshm\") pod \"5990b367-6bd6-483b-ab76-35375b015cbd\" (UID: \"5990b367-6bd6-483b-ab76-35375b015cbd\") " Oct 30 07:00:28.418740 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.418703 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-tmp-dir" pod "8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" (UID: "8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.419287 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.419177 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-model-cache" pod "8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" (UID: "8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.419415 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.419294 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-model-cache" pod "5990b367-6bd6-483b-ab76-35375b015cbd" (UID: "5990b367-6bd6-483b-ab76-35375b015cbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.419652 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.419620 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-tmp-dir" pod "5990b367-6bd6-483b-ab76-35375b015cbd" (UID: "5990b367-6bd6-483b-ab76-35375b015cbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.419751 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.419691 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-home" pod "8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" (UID: "8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.419828 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.419774 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-home" pod "5990b367-6bd6-483b-ab76-35375b015cbd" (UID: "5990b367-6bd6-483b-ab76-35375b015cbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.419931 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.419907 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.419997 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.419935 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.419997 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.419963 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.419997 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.419976 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.419997 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.419988 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.420416 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.420002 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.421368 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.421296 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-dshm" pod "8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" (UID: "8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.421652 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.421621 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-tls-certs" pod "8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" (UID: "8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:28.422496 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.422422 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-dshm" pod "5990b367-6bd6-483b-ab76-35375b015cbd" (UID: "5990b367-6bd6-483b-ab76-35375b015cbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:28.422579 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.422514 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5990b367-6bd6-483b-ab76-35375b015cbd-tls-certs" pod "5990b367-6bd6-483b-ab76-35375b015cbd" (UID: "5990b367-6bd6-483b-ab76-35375b015cbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:28.423022 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.422993 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-kube-api-access-qwvkf" pod "8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" (UID: "8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3"). InnerVolumeSpecName "kube-api-access-qwvkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:28.424024 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.423992 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5990b367-6bd6-483b-ab76-35375b015cbd-kube-api-access-h9d8k" pod "5990b367-6bd6-483b-ab76-35375b015cbd" (UID: "5990b367-6bd6-483b-ab76-35375b015cbd"). InnerVolumeSpecName "kube-api-access-h9d8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:28.521522 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.521493 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5990b367-6bd6-483b-ab76-35375b015cbd-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.521522 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.521520 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwvkf\" (UniqueName: \"kubernetes.io/projected/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-kube-api-access-qwvkf\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.521722 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.521535 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9d8k\" (UniqueName: \"kubernetes.io/projected/5990b367-6bd6-483b-ab76-35375b015cbd-kube-api-access-h9d8k\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.521722 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.521549 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5990b367-6bd6-483b-ab76-35375b015cbd-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.521722 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.521561 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.521722 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.521571 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:28.814344 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:28.814186 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:29.206886 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.206854 6999 generic.go:356] "Generic (PLEG): container finished" podID="6e53ae74-7c02-4cec-90e6-c5db61897c64" containerID="621ac90be0dae04d39af189d2dfab408baf7197b2849f80cec07b154c186353a" exitCode=0 Oct 30 07:00:29.207327 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.206938 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" event={"ID":"6e53ae74-7c02-4cec-90e6-c5db61897c64","Type":"ContainerDied","Data":"621ac90be0dae04d39af189d2dfab408baf7197b2849f80cec07b154c186353a"} Oct 30 07:00:29.208400 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.208383 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" Oct 30 07:00:29.208400 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.208389 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx" event={"ID":"8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3","Type":"ContainerDied","Data":"9463bceb8812bf4bec64eaf27a75bb91d7bb4616e418540bf7aa8b8e9115cbf1"} Oct 30 07:00:29.208555 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.208425 6999 scope.go:122] "RemoveContainer" containerID="405773f083aa8e5b74581b4f6ffd6f71a16028b93098ec9f1c3698b2786372dc" Oct 30 07:00:29.209974 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.209954 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" Oct 30 07:00:29.210178 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.210052 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb" event={"ID":"5990b367-6bd6-483b-ab76-35375b015cbd","Type":"ContainerDied","Data":"3a075484a4a33e3bb108552bb37dc425f212be7d917c6597d3e6dc690be5f9f5"} Oct 30 07:00:29.219607 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.219588 6999 scope.go:122] "RemoveContainer" containerID="fe0eff16cc7e522dc908c92eecc7e21c73811dc06956cbddc6906e8009907acf" Oct 30 07:00:29.282300 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.282272 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx"] Oct 30 07:00:29.283648 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.283626 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/simulator-w2-aaf467-kserve-6d8f69c99b-55hbx"] Oct 30 07:00:29.318107 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.318074 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb"] Oct 30 07:00:29.322282 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.322259 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-aaf467/distinct-w2-aaf467-kserve-987d7768d-hjkcb"] Oct 30 07:00:29.362957 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.362937 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" Oct 30 07:00:29.430662 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.430629 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d03989d-f895-4557-b293-4ae0313c0cc2-kube-api-access-lx7rm\" (UniqueName: \"kubernetes.io/projected/7d03989d-f895-4557-b293-4ae0313c0cc2-kube-api-access-lx7rm\") pod \"7d03989d-f895-4557-b293-4ae0313c0cc2\" (UID: \"7d03989d-f895-4557-b293-4ae0313c0cc2\") " Oct 30 07:00:29.430803 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.430729 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d03989d-f895-4557-b293-4ae0313c0cc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d03989d-f895-4557-b293-4ae0313c0cc2-maas-api-openshift-ca\") pod \"7d03989d-f895-4557-b293-4ae0313c0cc2\" (UID: \"7d03989d-f895-4557-b293-4ae0313c0cc2\") " Oct 30 07:00:29.430857 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.430840 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d03989d-f895-4557-b293-4ae0313c0cc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d03989d-f895-4557-b293-4ae0313c0cc2-maas-api-xks-ca\") pod \"7d03989d-f895-4557-b293-4ae0313c0cc2\" (UID: \"7d03989d-f895-4557-b293-4ae0313c0cc2\") " Oct 30 07:00:29.431147 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.431099 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d03989d-f895-4557-b293-4ae0313c0cc2-maas-api-openshift-ca" pod "7d03989d-f895-4557-b293-4ae0313c0cc2" (UID: "7d03989d-f895-4557-b293-4ae0313c0cc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:29.431362 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.431315 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d03989d-f895-4557-b293-4ae0313c0cc2-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:29.432942 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.432918 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d03989d-f895-4557-b293-4ae0313c0cc2-kube-api-access-lx7rm" pod "7d03989d-f895-4557-b293-4ae0313c0cc2" (UID: "7d03989d-f895-4557-b293-4ae0313c0cc2"). InnerVolumeSpecName "kube-api-access-lx7rm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:29.433128 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.433113 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d03989d-f895-4557-b293-4ae0313c0cc2-maas-api-xks-ca" pod "7d03989d-f895-4557-b293-4ae0313c0cc2" (UID: "7d03989d-f895-4557-b293-4ae0313c0cc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:29.532722 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.532664 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx7rm\" (UniqueName: \"kubernetes.io/projected/7d03989d-f895-4557-b293-4ae0313c0cc2-kube-api-access-lx7rm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:29.532722 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.532688 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d03989d-f895-4557-b293-4ae0313c0cc2-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:29.613117 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.613081 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5990b367-6bd6-483b-ab76-35375b015cbd" path="/var/lib/kubelet/pods/5990b367-6bd6-483b-ab76-35375b015cbd/volumes" Oct 30 07:00:29.613516 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:29.613503 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" path="/var/lib/kubelet/pods/8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3/volumes" Oct 30 07:00:30.218820 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:30.218792 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" Oct 30 07:00:30.219240 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:30.218828 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e97edfa7-7jjcs" event={"ID":"7d03989d-f895-4557-b293-4ae0313c0cc2","Type":"ContainerDied","Data":"f4c36798093a37320b655209b873e5d92b961aae2725918fe592372e4e7b2a47"} Oct 30 07:00:30.219240 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:30.218874 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4c36798093a37320b655209b873e5d92b961aae2725918fe592372e4e7b2a47" Oct 30 07:00:30.353746 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:30.353713 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" Oct 30 07:00:30.439636 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:30.439605 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e53ae74-7c02-4cec-90e6-c5db61897c64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e53ae74-7c02-4cec-90e6-c5db61897c64-maas-api-xks-ca\") pod \"6e53ae74-7c02-4cec-90e6-c5db61897c64\" (UID: \"6e53ae74-7c02-4cec-90e6-c5db61897c64\") " Oct 30 07:00:30.439802 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:30.439658 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e53ae74-7c02-4cec-90e6-c5db61897c64-kube-api-access-dvp6n\" (UniqueName: \"kubernetes.io/projected/6e53ae74-7c02-4cec-90e6-c5db61897c64-kube-api-access-dvp6n\") pod \"6e53ae74-7c02-4cec-90e6-c5db61897c64\" (UID: \"6e53ae74-7c02-4cec-90e6-c5db61897c64\") " Oct 30 07:00:30.439802 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:30.439774 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e53ae74-7c02-4cec-90e6-c5db61897c64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e53ae74-7c02-4cec-90e6-c5db61897c64-maas-api-openshift-ca\") pod \"6e53ae74-7c02-4cec-90e6-c5db61897c64\" (UID: \"6e53ae74-7c02-4cec-90e6-c5db61897c64\") " Oct 30 07:00:30.440104 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:30.440079 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e53ae74-7c02-4cec-90e6-c5db61897c64-maas-api-openshift-ca" pod "6e53ae74-7c02-4cec-90e6-c5db61897c64" (UID: "6e53ae74-7c02-4cec-90e6-c5db61897c64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:30.441972 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:30.441945 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e53ae74-7c02-4cec-90e6-c5db61897c64-maas-api-xks-ca" pod "6e53ae74-7c02-4cec-90e6-c5db61897c64" (UID: "6e53ae74-7c02-4cec-90e6-c5db61897c64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:30.442060 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:30.442046 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e53ae74-7c02-4cec-90e6-c5db61897c64-kube-api-access-dvp6n" pod "6e53ae74-7c02-4cec-90e6-c5db61897c64" (UID: "6e53ae74-7c02-4cec-90e6-c5db61897c64"). InnerVolumeSpecName "kube-api-access-dvp6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:30.540771 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:30.540716 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e53ae74-7c02-4cec-90e6-c5db61897c64-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:30.540771 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:30.540740 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvp6n\" (UniqueName: \"kubernetes.io/projected/6e53ae74-7c02-4cec-90e6-c5db61897c64-kube-api-access-dvp6n\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:30.540771 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:30.540750 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e53ae74-7c02-4cec-90e6-c5db61897c64-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:30.727353 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:30.727324 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:31.225905 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.225867 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" event={"ID":"6e53ae74-7c02-4cec-90e6-c5db61897c64","Type":"ContainerDied","Data":"908fd8988b380cc9839ed5611c3aff4156242d8fb66c7a533d5feabe7a30d385"} Oct 30 07:00:31.226336 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.225923 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="908fd8988b380cc9839ed5611c3aff4156242d8fb66c7a533d5feabe7a30d385" Oct 30 07:00:31.226336 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.225881 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g" Oct 30 07:00:31.462088 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.462051 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" podUID="987180e9-4669-4740-b2df-effebab90159" containerName="main" containerID="cri-o://0869967d3ebe554007b96692fc67efc53ab48de3cebf42b267fde945e67b8b2c" gracePeriod=45 Oct 30 07:00:31.715401 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.715374 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 07:00:31.753480 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.753416 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-model-cache\") pod \"987180e9-4669-4740-b2df-effebab90159\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " Oct 30 07:00:31.753480 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.753474 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-dshm\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-dshm\") pod \"987180e9-4669-4740-b2df-effebab90159\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " Oct 30 07:00:31.753689 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.753552 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/987180e9-4669-4740-b2df-effebab90159-tls-certs\" (UniqueName: \"kubernetes.io/secret/987180e9-4669-4740-b2df-effebab90159-tls-certs\") pod \"987180e9-4669-4740-b2df-effebab90159\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " Oct 30 07:00:31.753689 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.753627 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-home\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-home\") pod \"987180e9-4669-4740-b2df-effebab90159\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " Oct 30 07:00:31.753689 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.753652 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-model-cache" pod "987180e9-4669-4740-b2df-effebab90159" (UID: "987180e9-4669-4740-b2df-effebab90159"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.753689 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.753670 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-tmp-dir\") pod \"987180e9-4669-4740-b2df-effebab90159\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " Oct 30 07:00:31.753911 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.753707 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987180e9-4669-4740-b2df-effebab90159-kube-api-access-f626l\" (UniqueName: \"kubernetes.io/projected/987180e9-4669-4740-b2df-effebab90159-kube-api-access-f626l\") pod \"987180e9-4669-4740-b2df-effebab90159\" (UID: \"987180e9-4669-4740-b2df-effebab90159\") " Oct 30 07:00:31.753911 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.753832 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-tmp-dir" pod "987180e9-4669-4740-b2df-effebab90159" (UID: "987180e9-4669-4740-b2df-effebab90159"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.754016 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.753903 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-home" pod "987180e9-4669-4740-b2df-effebab90159" (UID: "987180e9-4669-4740-b2df-effebab90159"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.754079 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.754039 6999 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-home\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.754079 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.754053 6999 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-tmp-dir\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.754079 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.754068 6999 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-model-cache\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.755827 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.755804 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-dshm" pod "987180e9-4669-4740-b2df-effebab90159" (UID: "987180e9-4669-4740-b2df-effebab90159"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:31.755932 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.755849 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/987180e9-4669-4740-b2df-effebab90159-tls-certs" pod "987180e9-4669-4740-b2df-effebab90159" (UID: "987180e9-4669-4740-b2df-effebab90159"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:31.755932 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.755888 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987180e9-4669-4740-b2df-effebab90159-kube-api-access-f626l" pod "987180e9-4669-4740-b2df-effebab90159" (UID: "987180e9-4669-4740-b2df-effebab90159"). InnerVolumeSpecName "kube-api-access-f626l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:31.836393 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.836361 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:31.854978 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.854945 6999 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/987180e9-4669-4740-b2df-effebab90159-dshm\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.854978 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.854973 6999 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/987180e9-4669-4740-b2df-effebab90159-tls-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:31.854978 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:31.854985 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f626l\" (UniqueName: \"kubernetes.io/projected/987180e9-4669-4740-b2df-effebab90159-kube-api-access-f626l\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:32.232893 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:32.232853 6999 generic.go:356] "Generic (PLEG): container finished" podID="987180e9-4669-4740-b2df-effebab90159" containerID="0869967d3ebe554007b96692fc67efc53ab48de3cebf42b267fde945e67b8b2c" exitCode=0 Oct 30 07:00:32.233401 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:32.232907 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" Oct 30 07:00:32.233401 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:32.232941 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" event={"ID":"987180e9-4669-4740-b2df-effebab90159","Type":"ContainerDied","Data":"0869967d3ebe554007b96692fc67efc53ab48de3cebf42b267fde945e67b8b2c"} Oct 30 07:00:32.233401 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:32.232986 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht" event={"ID":"987180e9-4669-4740-b2df-effebab90159","Type":"ContainerDied","Data":"8d621f4d7cea4e7bc584bfaca3b4cff1e385f7d6317146ec2b369f91b1f6f04c"} Oct 30 07:00:32.233401 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:32.233005 6999 scope.go:122] "RemoveContainer" containerID="0869967d3ebe554007b96692fc67efc53ab48de3cebf42b267fde945e67b8b2c" Oct 30 07:00:32.243697 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:32.243676 6999 scope.go:122] "RemoveContainer" containerID="0869967d3ebe554007b96692fc67efc53ab48de3cebf42b267fde945e67b8b2c" Oct 30 07:00:32.243974 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:00:32.243947 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0869967d3ebe554007b96692fc67efc53ab48de3cebf42b267fde945e67b8b2c\": container with ID starting with 0869967d3ebe554007b96692fc67efc53ab48de3cebf42b267fde945e67b8b2c not found: ID does not exist" containerID="0869967d3ebe554007b96692fc67efc53ab48de3cebf42b267fde945e67b8b2c" Oct 30 07:00:32.244042 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:32.243985 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0869967d3ebe554007b96692fc67efc53ab48de3cebf42b267fde945e67b8b2c"} err="failed to get container status \"0869967d3ebe554007b96692fc67efc53ab48de3cebf42b267fde945e67b8b2c\": rpc error: code = NotFound desc = could not find container \"0869967d3ebe554007b96692fc67efc53ab48de3cebf42b267fde945e67b8b2c\": container with ID starting with 0869967d3ebe554007b96692fc67efc53ab48de3cebf42b267fde945e67b8b2c not found: ID does not exist" Oct 30 07:00:32.284322 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:32.284284 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht"] Oct 30 07:00:32.296314 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:32.296286 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8d6349/distinct-w3-8d6349-kserve-657d68664b-hrwht"] Oct 30 07:00:32.631177 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:32.631128 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:33.606764 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:33.606734 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987180e9-4669-4740-b2df-effebab90159" path="/var/lib/kubelet/pods/987180e9-4669-4740-b2df-effebab90159/volumes" Oct 30 07:00:34.144785 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:00:34.144747 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 07:00:34.447894 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:34.447812 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:36.331075 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:36.331044 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:37.682960 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.682930 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:37.832541 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.832506 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:37.841483 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.841449 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt"] Oct 30 07:00:37.841892 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.841878 6999 cpu_manager.go:414] "containerMap: removing container" podUID="7d03989d-f895-4557-b293-4ae0313c0cc2" containerName="revoke-keys" Oct 30 07:00:37.841935 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.841894 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d03989d-f895-4557-b293-4ae0313c0cc2" containerName="revoke-keys" Oct 30 07:00:37.841935 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.841906 6999 cpu_manager.go:414] "containerMap: removing container" podUID="6e53ae74-7c02-4cec-90e6-c5db61897c64" containerName="revoke-keys" Oct 30 07:00:37.841935 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.841911 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e53ae74-7c02-4cec-90e6-c5db61897c64" containerName="revoke-keys" Oct 30 07:00:37.842037 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.841936 6999 cpu_manager.go:414] "containerMap: removing container" podUID="5990b367-6bd6-483b-ab76-35375b015cbd" containerName="main" Oct 30 07:00:37.842037 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.841941 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5990b367-6bd6-483b-ab76-35375b015cbd" containerName="main" Oct 30 07:00:37.842037 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.841950 6999 cpu_manager.go:414] "containerMap: removing container" podUID="987180e9-4669-4740-b2df-effebab90159" containerName="main" Oct 30 07:00:37.842037 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.841955 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987180e9-4669-4740-b2df-effebab90159" containerName="main" Oct 30 07:00:37.842037 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.841965 6999 cpu_manager.go:414] "containerMap: removing container" podUID="8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" containerName="main" Oct 30 07:00:37.842037 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.841969 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" containerName="main" Oct 30 07:00:37.842037 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.841979 6999 cpu_manager.go:414] "containerMap: removing container" podUID="5acb488c-537b-4fdc-9555-246b327de37d" containerName="revoke-keys" Oct 30 07:00:37.842037 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.841983 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5acb488c-537b-4fdc-9555-246b327de37d" containerName="revoke-keys" Oct 30 07:00:37.842314 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.842043 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d03989d-f895-4557-b293-4ae0313c0cc2" containerName="revoke-keys" Oct 30 07:00:37.842314 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.842049 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e53ae74-7c02-4cec-90e6-c5db61897c64" containerName="revoke-keys" Oct 30 07:00:37.842314 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.842062 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="5990b367-6bd6-483b-ab76-35375b015cbd" containerName="main" Oct 30 07:00:37.842314 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.842068 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="987180e9-4669-4740-b2df-effebab90159" containerName="main" Oct 30 07:00:37.842314 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.842075 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e6a2ea8-fe55-4e70-bf40-664ce2ba1ce3" containerName="main" Oct 30 07:00:37.842314 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.842080 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="5acb488c-537b-4fdc-9555-246b327de37d" containerName="revoke-keys" Oct 30 07:00:37.846408 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.846391 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" Oct 30 07:00:37.874305 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.874279 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 07:00:37.874461 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.874439 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zh9cj\"" Oct 30 07:00:37.876749 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.876727 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt"] Oct 30 07:00:37.909862 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.909825 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02725f55-6ed2-4181-948e-4acf183e6e45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60f9a89b-q6clt\" (UID: \"02725f55-6ed2-4181-948e-4acf183e6e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" Oct 30 07:00:37.910013 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.909981 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pkkl\" (UniqueName: \"kubernetes.io/projected/02725f55-6ed2-4181-948e-4acf183e6e45-kube-api-access-8pkkl\") pod \"maas-api-revoke-sub-60f9a89b-q6clt\" (UID: \"02725f55-6ed2-4181-948e-4acf183e6e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" Oct 30 07:00:37.910095 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.910039 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02725f55-6ed2-4181-948e-4acf183e6e45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60f9a89b-q6clt\" (UID: \"02725f55-6ed2-4181-948e-4acf183e6e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" Oct 30 07:00:37.999883 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:37.999803 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:38.011370 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:38.011342 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pkkl\" (UniqueName: \"kubernetes.io/projected/02725f55-6ed2-4181-948e-4acf183e6e45-kube-api-access-8pkkl\") pod \"maas-api-revoke-sub-60f9a89b-q6clt\" (UID: \"02725f55-6ed2-4181-948e-4acf183e6e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" Oct 30 07:00:38.011529 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:38.011386 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02725f55-6ed2-4181-948e-4acf183e6e45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60f9a89b-q6clt\" (UID: \"02725f55-6ed2-4181-948e-4acf183e6e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" Oct 30 07:00:38.011529 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:38.011440 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02725f55-6ed2-4181-948e-4acf183e6e45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60f9a89b-q6clt\" (UID: \"02725f55-6ed2-4181-948e-4acf183e6e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" Oct 30 07:00:38.012135 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:38.012099 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02725f55-6ed2-4181-948e-4acf183e6e45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60f9a89b-q6clt\" (UID: \"02725f55-6ed2-4181-948e-4acf183e6e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" Oct 30 07:00:38.014042 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:38.014017 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02725f55-6ed2-4181-948e-4acf183e6e45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60f9a89b-q6clt\" (UID: \"02725f55-6ed2-4181-948e-4acf183e6e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" Oct 30 07:00:38.025229 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:38.025174 6999 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" podUID="f977329f-2b08-42f3-9359-154994886be9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 07:00:38.034827 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:38.034795 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pkkl\" (UniqueName: \"kubernetes.io/projected/02725f55-6ed2-4181-948e-4acf183e6e45-kube-api-access-8pkkl\") pod \"maas-api-revoke-sub-60f9a89b-q6clt\" (UID: \"02725f55-6ed2-4181-948e-4acf183e6e45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" Oct 30 07:00:38.160942 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:38.160905 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" Oct 30 07:00:38.466509 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:38.466480 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt"] Oct 30 07:00:38.468550 ip-10-0-131-226 kubenswrapper[6999]: W1030 07:00:38.468513 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02725f55_6ed2_4181_948e_4acf183e6e45.slice/crio-20353853c5c83d9510c0fb41ff33763c8aee1df954be0b1e393b6d0bb821aabe WatchSource:0}: Error finding container 20353853c5c83d9510c0fb41ff33763c8aee1df954be0b1e393b6d0bb821aabe: Status 404 returned error can't find the container with id 20353853c5c83d9510c0fb41ff33763c8aee1df954be0b1e393b6d0bb821aabe Oct 30 07:00:38.513938 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:38.513911 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:38.625566 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:38.625530 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9"] Oct 30 07:00:38.628475 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:38.628450 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8d6349-bhdr9"] Oct 30 07:00:38.664457 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:38.664337 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:39.175837 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:39.175803 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:39.273712 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:39.273687 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" event={"ID":"02725f55-6ed2-4181-948e-4acf183e6e45","Type":"ContainerStarted","Data":"4b09f36f4ced08d99eadb288083e81bfc85c5dec3f3cc057614af613b7191958"} Oct 30 07:00:39.273842 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:39.273734 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" event={"ID":"02725f55-6ed2-4181-948e-4acf183e6e45","Type":"ContainerStarted","Data":"20353853c5c83d9510c0fb41ff33763c8aee1df954be0b1e393b6d0bb821aabe"} Oct 30 07:00:39.313972 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:39.313915 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" podStartSLOduration=2.313902055 podStartE2EDuration="2.313902055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 07:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:00:39.31340911 +0000 UTC m=+4005.822428997" watchObservedRunningTime="2026-10-30 07:00:39.313902055 +0000 UTC m=+4005.822921936" Oct 30 07:00:39.446392 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:39.446370 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" Oct 30 07:00:39.527140 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:39.527113 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f977329f-2b08-42f3-9359-154994886be9-kube-api-access-zpdxb\" (UniqueName: \"kubernetes.io/projected/f977329f-2b08-42f3-9359-154994886be9-kube-api-access-zpdxb\") pod \"f977329f-2b08-42f3-9359-154994886be9\" (UID: \"f977329f-2b08-42f3-9359-154994886be9\") " Oct 30 07:00:39.527364 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:39.527172 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f977329f-2b08-42f3-9359-154994886be9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f977329f-2b08-42f3-9359-154994886be9-plugins-config-volume\") pod \"f977329f-2b08-42f3-9359-154994886be9\" (UID: \"f977329f-2b08-42f3-9359-154994886be9\") " Oct 30 07:00:39.527537 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:39.527512 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f977329f-2b08-42f3-9359-154994886be9-plugins-config-volume" pod "f977329f-2b08-42f3-9359-154994886be9" (UID: "f977329f-2b08-42f3-9359-154994886be9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:39.529382 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:39.529363 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f977329f-2b08-42f3-9359-154994886be9-kube-api-access-zpdxb" pod "f977329f-2b08-42f3-9359-154994886be9" (UID: "f977329f-2b08-42f3-9359-154994886be9"). InnerVolumeSpecName "kube-api-access-zpdxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:39.607708 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:39.607672 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="256d51fb-6115-40df-aa29-35fcde48dfb3" path="/var/lib/kubelet/pods/256d51fb-6115-40df-aa29-35fcde48dfb3/volumes" Oct 30 07:00:39.628617 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:39.628597 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpdxb\" (UniqueName: \"kubernetes.io/projected/f977329f-2b08-42f3-9359-154994886be9-kube-api-access-zpdxb\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:39.628617 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:39.628618 6999 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f977329f-2b08-42f3-9359-154994886be9-plugins-config-volume\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:40.063736 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.063699 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:40.280390 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.280313 6999 generic.go:356] "Generic (PLEG): container finished" podID="f977329f-2b08-42f3-9359-154994886be9" containerID="a9f73eeadaa81e4433aa93402a85a2648d6f9e3fbb882eeb7b22cbfcb8ad85ab" exitCode=0 Oct 30 07:00:40.280390 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.280371 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" Oct 30 07:00:40.280919 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.280385 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" event={"ID":"f977329f-2b08-42f3-9359-154994886be9","Type":"ContainerDied","Data":"a9f73eeadaa81e4433aa93402a85a2648d6f9e3fbb882eeb7b22cbfcb8ad85ab"} Oct 30 07:00:40.280919 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.280457 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq" event={"ID":"f977329f-2b08-42f3-9359-154994886be9","Type":"ContainerDied","Data":"5108ab9515ce6666c640e1353f61ea9cecfb475cb93007cfbd398a5271d18b17"} Oct 30 07:00:40.280919 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.280481 6999 scope.go:122] "RemoveContainer" containerID="a9f73eeadaa81e4433aa93402a85a2648d6f9e3fbb882eeb7b22cbfcb8ad85ab" Oct 30 07:00:40.283238 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.283185 6999 generic.go:356] "Generic (PLEG): container finished" podID="02725f55-6ed2-4181-948e-4acf183e6e45" containerID="4b09f36f4ced08d99eadb288083e81bfc85c5dec3f3cc057614af613b7191958" exitCode=0 Oct 30 07:00:40.283369 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.283274 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" event={"ID":"02725f55-6ed2-4181-948e-4acf183e6e45","Type":"ContainerDied","Data":"4b09f36f4ced08d99eadb288083e81bfc85c5dec3f3cc057614af613b7191958"} Oct 30 07:00:40.290050 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.290034 6999 scope.go:122] "RemoveContainer" containerID="a9f73eeadaa81e4433aa93402a85a2648d6f9e3fbb882eeb7b22cbfcb8ad85ab" Oct 30 07:00:40.290417 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:00:40.290378 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f73eeadaa81e4433aa93402a85a2648d6f9e3fbb882eeb7b22cbfcb8ad85ab\": container with ID starting with a9f73eeadaa81e4433aa93402a85a2648d6f9e3fbb882eeb7b22cbfcb8ad85ab not found: ID does not exist" containerID="a9f73eeadaa81e4433aa93402a85a2648d6f9e3fbb882eeb7b22cbfcb8ad85ab" Oct 30 07:00:40.290536 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.290428 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f73eeadaa81e4433aa93402a85a2648d6f9e3fbb882eeb7b22cbfcb8ad85ab"} err="failed to get container status \"a9f73eeadaa81e4433aa93402a85a2648d6f9e3fbb882eeb7b22cbfcb8ad85ab\": rpc error: code = NotFound desc = could not find container \"a9f73eeadaa81e4433aa93402a85a2648d6f9e3fbb882eeb7b22cbfcb8ad85ab\": container with ID starting with a9f73eeadaa81e4433aa93402a85a2648d6f9e3fbb882eeb7b22cbfcb8ad85ab not found: ID does not exist" Oct 30 07:00:40.370899 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.370874 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq"] Oct 30 07:00:40.377581 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.377553 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-fa5a69-5884494dcf-vgdqq"] Oct 30 07:00:40.568057 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.567976 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:40.684492 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.684461 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj"] Oct 30 07:00:40.689591 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.689553 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-aaf467-7mvgj"] Oct 30 07:00:40.713057 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:40.713029 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:41.236379 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:41.236344 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:41.427360 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:41.427339 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" Oct 30 07:00:41.550273 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:41.550188 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02725f55-6ed2-4181-948e-4acf183e6e45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02725f55-6ed2-4181-948e-4acf183e6e45-maas-api-xks-ca\") pod \"02725f55-6ed2-4181-948e-4acf183e6e45\" (UID: \"02725f55-6ed2-4181-948e-4acf183e6e45\") " Oct 30 07:00:41.550421 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:41.550289 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02725f55-6ed2-4181-948e-4acf183e6e45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02725f55-6ed2-4181-948e-4acf183e6e45-maas-api-openshift-ca\") pod \"02725f55-6ed2-4181-948e-4acf183e6e45\" (UID: \"02725f55-6ed2-4181-948e-4acf183e6e45\") " Oct 30 07:00:41.550421 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:41.550374 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02725f55-6ed2-4181-948e-4acf183e6e45-kube-api-access-8pkkl\" (UniqueName: \"kubernetes.io/projected/02725f55-6ed2-4181-948e-4acf183e6e45-kube-api-access-8pkkl\") pod \"02725f55-6ed2-4181-948e-4acf183e6e45\" (UID: \"02725f55-6ed2-4181-948e-4acf183e6e45\") " Oct 30 07:00:41.550670 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:41.550647 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02725f55-6ed2-4181-948e-4acf183e6e45-maas-api-openshift-ca" pod "02725f55-6ed2-4181-948e-4acf183e6e45" (UID: "02725f55-6ed2-4181-948e-4acf183e6e45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:41.552615 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:41.552585 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02725f55-6ed2-4181-948e-4acf183e6e45-maas-api-xks-ca" pod "02725f55-6ed2-4181-948e-4acf183e6e45" (UID: "02725f55-6ed2-4181-948e-4acf183e6e45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 07:00:41.552615 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:41.552607 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02725f55-6ed2-4181-948e-4acf183e6e45-kube-api-access-8pkkl" pod "02725f55-6ed2-4181-948e-4acf183e6e45" (UID: "02725f55-6ed2-4181-948e-4acf183e6e45"). InnerVolumeSpecName "kube-api-access-8pkkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:41.607083 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:41.607054 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="108a685d-1c86-402f-914f-d7574a1deef5" path="/var/lib/kubelet/pods/108a685d-1c86-402f-914f-d7574a1deef5/volumes" Oct 30 07:00:41.607480 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:41.607465 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f977329f-2b08-42f3-9359-154994886be9" path="/var/lib/kubelet/pods/f977329f-2b08-42f3-9359-154994886be9/volumes" Oct 30 07:00:41.651263 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:41.651237 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02725f55-6ed2-4181-948e-4acf183e6e45-maas-api-openshift-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:41.651263 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:41.651258 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pkkl\" (UniqueName: \"kubernetes.io/projected/02725f55-6ed2-4181-948e-4acf183e6e45-kube-api-access-8pkkl\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:41.651414 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:41.651268 6999 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02725f55-6ed2-4181-948e-4acf183e6e45-maas-api-xks-ca\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:42.295992 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:42.295960 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" Oct 30 07:00:42.295992 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:42.295984 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60f9a89b-q6clt" event={"ID":"02725f55-6ed2-4181-948e-4acf183e6e45","Type":"ContainerDied","Data":"20353853c5c83d9510c0fb41ff33763c8aee1df954be0b1e393b6d0bb821aabe"} Oct 30 07:00:42.296324 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:42.296027 6999 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20353853c5c83d9510c0fb41ff33763c8aee1df954be0b1e393b6d0bb821aabe" Oct 30 07:00:46.340655 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:46.340626 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:46.474302 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:46.474265 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:46.753874 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:46.753833 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:49.099771 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:49.099702 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:49.245640 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:49.245606 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:49.372459 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:49.372426 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx"] Oct 30 07:00:49.372676 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:49.372654 6999 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" podUID="eaa497a5-ee9b-4672-b80b-914b3ac99692" containerName="istio-proxy" containerID="cri-o://31a4023a9e5b3832b2bf5999a37f969a867e977e9f159df6c15d518a7d602ab0" gracePeriod=30 Oct 30 07:00:49.448289 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:49.448259 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:50.302464 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:50.302413 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:50.435738 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:50.435706 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:51.313701 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:51.313671 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:52.389661 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:52.389628 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:52.894963 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:52.894926 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:53.018125 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:53.018095 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g"] Oct 30 07:00:53.029754 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:53.029723 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-8d44dd-m2d6g"] Oct 30 07:00:53.040569 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:53.040539 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:53.502898 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:53.502857 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:53.608928 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:53.608895 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e53ae74-7c02-4cec-90e6-c5db61897c64" path="/var/lib/kubelet/pods/6e53ae74-7c02-4cec-90e6-c5db61897c64/volumes" Oct 30 07:00:53.645715 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:53.645686 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:54.624234 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.624175 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 07:00:54.685069 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.684989 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:54.776273 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.776244 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-data\") pod \"eaa497a5-ee9b-4672-b80b-914b3ac99692\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " Oct 30 07:00:54.776452 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.776302 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-podinfo\") pod \"eaa497a5-ee9b-4672-b80b-914b3ac99692\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " Oct 30 07:00:54.776452 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.776325 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-token\" (UniqueName: \"kubernetes.io/projected/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-token\") pod \"eaa497a5-ee9b-4672-b80b-914b3ac99692\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " Oct 30 07:00:54.776452 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.776358 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa497a5-ee9b-4672-b80b-914b3ac99692-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa497a5-ee9b-4672-b80b-914b3ac99692-istiod-ca-cert\") pod \"eaa497a5-ee9b-4672-b80b-914b3ac99692\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " Oct 30 07:00:54.776452 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.776395 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-envoy\") pod \"eaa497a5-ee9b-4672-b80b-914b3ac99692\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " Oct 30 07:00:54.776694 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.776455 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-workload-socket\") pod \"eaa497a5-ee9b-4672-b80b-914b3ac99692\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " Oct 30 07:00:54.776694 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.776517 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa497a5-ee9b-4672-b80b-914b3ac99692-kube-api-access-d8pj8\" (UniqueName: \"kubernetes.io/projected/eaa497a5-ee9b-4672-b80b-914b3ac99692-kube-api-access-d8pj8\") pod \"eaa497a5-ee9b-4672-b80b-914b3ac99692\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " Oct 30 07:00:54.776694 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.776533 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-data" pod "eaa497a5-ee9b-4672-b80b-914b3ac99692" (UID: "eaa497a5-ee9b-4672-b80b-914b3ac99692"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:54.776694 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.776582 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-credential-socket\") pod \"eaa497a5-ee9b-4672-b80b-914b3ac99692\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " Oct 30 07:00:54.776922 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.776802 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-workload-socket" pod "eaa497a5-ee9b-4672-b80b-914b3ac99692" (UID: "eaa497a5-ee9b-4672-b80b-914b3ac99692"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:54.776922 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.776845 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-credential-socket" pod "eaa497a5-ee9b-4672-b80b-914b3ac99692" (UID: "eaa497a5-ee9b-4672-b80b-914b3ac99692"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:54.776922 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.776873 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa497a5-ee9b-4672-b80b-914b3ac99692-istiod-ca-cert" pod "eaa497a5-ee9b-4672-b80b-914b3ac99692" (UID: "eaa497a5-ee9b-4672-b80b-914b3ac99692"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 07:00:54.777110 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.776610 6999 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-workload-certs\") pod \"eaa497a5-ee9b-4672-b80b-914b3ac99692\" (UID: \"eaa497a5-ee9b-4672-b80b-914b3ac99692\") " Oct 30 07:00:54.777173 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.777073 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-workload-certs" pod "eaa497a5-ee9b-4672-b80b-914b3ac99692" (UID: "eaa497a5-ee9b-4672-b80b-914b3ac99692"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:54.777293 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.777267 6999 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa497a5-ee9b-4672-b80b-914b3ac99692-istiod-ca-cert\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:54.777433 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.777317 6999 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-workload-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:54.777433 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.777327 6999 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-credential-socket\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:54.777433 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.777336 6999 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-data\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:54.778867 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.778839 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-token" pod "eaa497a5-ee9b-4672-b80b-914b3ac99692" (UID: "eaa497a5-ee9b-4672-b80b-914b3ac99692"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:54.778955 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.778920 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa497a5-ee9b-4672-b80b-914b3ac99692-kube-api-access-d8pj8" pod "eaa497a5-ee9b-4672-b80b-914b3ac99692" (UID: "eaa497a5-ee9b-4672-b80b-914b3ac99692"). InnerVolumeSpecName "kube-api-access-d8pj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 07:00:54.779101 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.779084 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-podinfo" pod "eaa497a5-ee9b-4672-b80b-914b3ac99692" (UID: "eaa497a5-ee9b-4672-b80b-914b3ac99692"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 07:00:54.779144 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.779095 6999 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-envoy" pod "eaa497a5-ee9b-4672-b80b-914b3ac99692" (UID: "eaa497a5-ee9b-4672-b80b-914b3ac99692"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 07:00:54.878320 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.878294 6999 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-token\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:54.878320 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.878317 6999 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-envoy\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:54.878475 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.878326 6999 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8pj8\" (UniqueName: \"kubernetes.io/projected/eaa497a5-ee9b-4672-b80b-914b3ac99692-kube-api-access-d8pj8\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:54.878475 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.878335 6999 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa497a5-ee9b-4672-b80b-914b3ac99692-workload-certs\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:54.878475 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:54.878345 6999 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa497a5-ee9b-4672-b80b-914b3ac99692-istio-podinfo\") on node \"ip-10-0-131-226.ec2.internal\" DevicePath \"\"" Oct 30 07:00:55.366385 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:55.366353 6999 generic.go:356] "Generic (PLEG): container finished" podID="eaa497a5-ee9b-4672-b80b-914b3ac99692" containerID="31a4023a9e5b3832b2bf5999a37f969a867e977e9f159df6c15d518a7d602ab0" exitCode=0 Oct 30 07:00:55.366571 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:55.366397 6999 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" Oct 30 07:00:55.366571 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:55.366442 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" event={"ID":"eaa497a5-ee9b-4672-b80b-914b3ac99692","Type":"ContainerDied","Data":"31a4023a9e5b3832b2bf5999a37f969a867e977e9f159df6c15d518a7d602ab0"} Oct 30 07:00:55.366571 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:55.366497 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx" event={"ID":"eaa497a5-ee9b-4672-b80b-914b3ac99692","Type":"ContainerDied","Data":"45e0ab622a54c29ed2b25416d1a7e4016fc233577889ab6c5914dbb0d3971751"} Oct 30 07:00:55.366571 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:55.366521 6999 scope.go:122] "RemoveContainer" containerID="31a4023a9e5b3832b2bf5999a37f969a867e977e9f159df6c15d518a7d602ab0" Oct 30 07:00:55.376558 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:55.376543 6999 scope.go:122] "RemoveContainer" containerID="31a4023a9e5b3832b2bf5999a37f969a867e977e9f159df6c15d518a7d602ab0" Oct 30 07:00:55.376812 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:00:55.376795 6999 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31a4023a9e5b3832b2bf5999a37f969a867e977e9f159df6c15d518a7d602ab0\": container with ID starting with 31a4023a9e5b3832b2bf5999a37f969a867e977e9f159df6c15d518a7d602ab0 not found: ID does not exist" containerID="31a4023a9e5b3832b2bf5999a37f969a867e977e9f159df6c15d518a7d602ab0" Oct 30 07:00:55.376862 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:55.376820 6999 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31a4023a9e5b3832b2bf5999a37f969a867e977e9f159df6c15d518a7d602ab0"} err="failed to get container status \"31a4023a9e5b3832b2bf5999a37f969a867e977e9f159df6c15d518a7d602ab0\": rpc error: code = NotFound desc = could not find container \"31a4023a9e5b3832b2bf5999a37f969a867e977e9f159df6c15d518a7d602ab0\": container with ID starting with 31a4023a9e5b3832b2bf5999a37f969a867e977e9f159df6c15d518a7d602ab0 not found: ID does not exist" Oct 30 07:00:55.431041 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:55.431015 6999 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx"] Oct 30 07:00:55.436575 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:55.436553 6999 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aaf467-openshift-default-7f46d488c5-swkrx"] Oct 30 07:00:55.607467 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:55.607438 6999 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa497a5-ee9b-4672-b80b-914b3ac99692" path="/var/lib/kubelet/pods/eaa497a5-ee9b-4672-b80b-914b3ac99692/volumes" Oct 30 07:00:57.401385 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:57.401355 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:59.139524 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:59.139486 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:00:59.283305 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:00:59.283273 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:00.301360 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:00.301317 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:00.455756 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:00.455710 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:00.716968 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:00.716933 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:01.447730 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:01.447694 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:02.452720 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:02.452690 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-bwccj_013f3068-30a0-4b4c-b313-6eee3520eb14/kuadrant-console-plugin/0.log" Oct 30 07:01:04.142406 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:01:04.142372 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 07:01:05.001040 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:05.001011 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dd578_d2d51b73-b344-44d8-8fbf-84b50e45cbe5/manager/0.log" Oct 30 07:01:05.442404 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:05.442379 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5xns7_c2c64f13-36fe-4739-930b-f094dd0e225c/manager/0.log" Oct 30 07:01:05.911549 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:05.911514 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/1.log" Oct 30 07:01:06.107123 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:06.107093 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-wm62c_0e3d6cb4-b70b-4d68-aca8-53fdb36f017d/server/0.log" Oct 30 07:01:06.307007 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:06.306925 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kjczm_c65427c6-ca18-462d-b777-c52b0b195c9c/manager/0.log" Oct 30 07:01:06.518957 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:06.518930 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wmwrb_4d8da3c6-77af-4a68-aa92-89594aaa5037/proxy/0.log" Oct 30 07:01:06.910387 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:06.910358 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/0.log" Oct 30 07:01:06.930638 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:06.930611 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:08.031692 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:08.031662 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:08.173141 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:08.173101 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:09.433333 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:09.433303 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:10.294190 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:10.294160 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:16.175836 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:16.175804 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:19.009727 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:19.009695 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dd578_d2d51b73-b344-44d8-8fbf-84b50e45cbe5/manager/0.log" Oct 30 07:01:19.449366 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:19.449338 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5xns7_c2c64f13-36fe-4739-930b-f094dd0e225c/manager/0.log" Oct 30 07:01:19.929925 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:19.929895 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/1.log" Oct 30 07:01:20.118420 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:20.118394 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-wm62c_0e3d6cb4-b70b-4d68-aca8-53fdb36f017d/server/0.log" Oct 30 07:01:20.329722 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:20.329654 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kjczm_c65427c6-ca18-462d-b777-c52b0b195c9c/manager/0.log" Oct 30 07:01:20.557006 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:20.556975 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wmwrb_4d8da3c6-77af-4a68-aa92-89594aaa5037/proxy/0.log" Oct 30 07:01:20.975442 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:20.975402 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/0.log" Oct 30 07:01:22.204413 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:22.204380 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-60f9a89b-q6clt_02725f55-6ed2-4181-948e-4acf183e6e45/revoke-keys/0.log" Oct 30 07:01:23.921539 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:23.921509 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d589320d-qhkmq_5acb488c-537b-4fdc-9555-246b327de37d/revoke-keys/0.log" Oct 30 07:01:24.110739 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:24.110711 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e97edfa7-7jjcs_7d03989d-f895-4557-b293-4ae0313c0cc2/revoke-keys/0.log" Oct 30 07:01:24.498447 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:24.498424 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-skg82_f3f6f187-d56f-487a-a923-7d6d310a03d1/postgres/0.log" Oct 30 07:01:26.795819 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:26.795791 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-bwccj_013f3068-30a0-4b4c-b313-6eee3520eb14/kuadrant-console-plugin/0.log" Oct 30 07:01:29.682418 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:29.682385 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c4bb65c68-z6847_0d476729-9d6f-4624-b080-e19e28bcd0f5/kube-auth-proxy/0.log" Oct 30 07:01:30.099026 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:30.098947 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-mlqh8_edf39134-4250-4b1c-af52-f5d4f8ddcd10/istio-proxy/0.log" Oct 30 07:01:30.932883 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:30.932848 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:31.073229 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:31.073190 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:34.140817 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:01:34.140729 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 07:01:43.692160 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:43.692133 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:43.838136 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:43.838103 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:01:49.563048 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:49.563020 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pp7mt_a968117a-0683-4844-8936-99ef536894d9/global-pull-secret-syncer/0.log" Oct 30 07:01:50.231050 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:50.231017 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" Oct 30 07:01:50.854829 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:50.854797 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-226.ec2.internal_0b8548329cbac7d7fb7409929225464e/haproxy/0.log" Oct 30 07:01:58.414671 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:58.414641 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-bwccj_013f3068-30a0-4b4c-b313-6eee3520eb14/kuadrant-console-plugin/0.log" Oct 30 07:01:58.998131 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:58.998098 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dd578_d2d51b73-b344-44d8-8fbf-84b50e45cbe5/manager/0.log" Oct 30 07:01:59.676554 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.676509 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5xns7_c2c64f13-36fe-4739-930b-f094dd0e225c/manager/0.log" Oct 30 07:01:59.729010 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.728978 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/0.log" Oct 30 07:01:59.891628 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.891592 6999 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252"] Oct 30 07:01:59.892205 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.892184 6999 cpu_manager.go:414] "containerMap: removing container" podUID="eaa497a5-ee9b-4672-b80b-914b3ac99692" containerName="istio-proxy" Oct 30 07:01:59.892205 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.892226 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa497a5-ee9b-4672-b80b-914b3ac99692" containerName="istio-proxy" Oct 30 07:01:59.892412 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.892284 6999 cpu_manager.go:414] "containerMap: removing container" podUID="02725f55-6ed2-4181-948e-4acf183e6e45" containerName="revoke-keys" Oct 30 07:01:59.892412 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.892293 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02725f55-6ed2-4181-948e-4acf183e6e45" containerName="revoke-keys" Oct 30 07:01:59.892412 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.892307 6999 cpu_manager.go:414] "containerMap: removing container" podUID="f977329f-2b08-42f3-9359-154994886be9" containerName="payload-pre-processing" Oct 30 07:01:59.892412 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.892315 6999 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f977329f-2b08-42f3-9359-154994886be9" containerName="payload-pre-processing" Oct 30 07:01:59.892622 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.892425 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa497a5-ee9b-4672-b80b-914b3ac99692" containerName="istio-proxy" Oct 30 07:01:59.892622 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.892446 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="02725f55-6ed2-4181-948e-4acf183e6e45" containerName="revoke-keys" Oct 30 07:01:59.892622 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.892457 6999 memory_manager.go:365] "RemoveStaleState removing state" podUID="f977329f-2b08-42f3-9359-154994886be9" containerName="payload-pre-processing" Oct 30 07:01:59.895613 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.895594 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:01:59.913293 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.913272 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5nqnt\"/\"kube-root-ca.crt\"" Oct 30 07:01:59.925459 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.925438 6999 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5nqnt\"/\"openshift-service-ca.crt\"" Oct 30 07:01:59.925721 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.925697 6999 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5nqnt\"/\"default-dockercfg-w6ccv\"" Oct 30 07:01:59.927992 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.927916 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252"] Oct 30 07:01:59.955941 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.955914 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4af7e1f9-3e4c-4a48-8899-de031c15e1ec-sys\") pod \"perf-node-gather-daemonset-cl252\" (UID: \"4af7e1f9-3e4c-4a48-8899-de031c15e1ec\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:01:59.956058 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.955980 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4af7e1f9-3e4c-4a48-8899-de031c15e1ec-lib-modules\") pod \"perf-node-gather-daemonset-cl252\" (UID: \"4af7e1f9-3e4c-4a48-8899-de031c15e1ec\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:01:59.956102 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.956049 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4af7e1f9-3e4c-4a48-8899-de031c15e1ec-proc\") pod \"perf-node-gather-daemonset-cl252\" (UID: \"4af7e1f9-3e4c-4a48-8899-de031c15e1ec\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:01:59.956102 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.956098 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4af7e1f9-3e4c-4a48-8899-de031c15e1ec-podres\") pod \"perf-node-gather-daemonset-cl252\" (UID: \"4af7e1f9-3e4c-4a48-8899-de031c15e1ec\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:01:59.956173 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:01:59.956118 6999 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jggd\" (UniqueName: \"kubernetes.io/projected/4af7e1f9-3e4c-4a48-8899-de031c15e1ec-kube-api-access-9jggd\") pod \"perf-node-gather-daemonset-cl252\" (UID: \"4af7e1f9-3e4c-4a48-8899-de031c15e1ec\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:02:00.019555 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.019532 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/1.log" Oct 30 07:02:00.057487 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.057460 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4af7e1f9-3e4c-4a48-8899-de031c15e1ec-sys\") pod \"perf-node-gather-daemonset-cl252\" (UID: \"4af7e1f9-3e4c-4a48-8899-de031c15e1ec\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:02:00.057631 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.057503 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4af7e1f9-3e4c-4a48-8899-de031c15e1ec-lib-modules\") pod \"perf-node-gather-daemonset-cl252\" (UID: \"4af7e1f9-3e4c-4a48-8899-de031c15e1ec\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:02:00.057631 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.057522 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4af7e1f9-3e4c-4a48-8899-de031c15e1ec-proc\") pod \"perf-node-gather-daemonset-cl252\" (UID: \"4af7e1f9-3e4c-4a48-8899-de031c15e1ec\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:02:00.057631 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.057584 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4af7e1f9-3e4c-4a48-8899-de031c15e1ec-podres\") pod \"perf-node-gather-daemonset-cl252\" (UID: \"4af7e1f9-3e4c-4a48-8899-de031c15e1ec\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:02:00.057631 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.057591 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4af7e1f9-3e4c-4a48-8899-de031c15e1ec-sys\") pod \"perf-node-gather-daemonset-cl252\" (UID: \"4af7e1f9-3e4c-4a48-8899-de031c15e1ec\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:02:00.057631 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.057610 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4af7e1f9-3e4c-4a48-8899-de031c15e1ec-lib-modules\") pod \"perf-node-gather-daemonset-cl252\" (UID: \"4af7e1f9-3e4c-4a48-8899-de031c15e1ec\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:02:00.057631 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.057598 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4af7e1f9-3e4c-4a48-8899-de031c15e1ec-proc\") pod \"perf-node-gather-daemonset-cl252\" (UID: \"4af7e1f9-3e4c-4a48-8899-de031c15e1ec\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:02:00.057962 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.057643 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4af7e1f9-3e4c-4a48-8899-de031c15e1ec-podres\") pod \"perf-node-gather-daemonset-cl252\" (UID: \"4af7e1f9-3e4c-4a48-8899-de031c15e1ec\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:02:00.057962 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.057641 6999 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jggd\" (UniqueName: \"kubernetes.io/projected/4af7e1f9-3e4c-4a48-8899-de031c15e1ec-kube-api-access-9jggd\") pod \"perf-node-gather-daemonset-cl252\" (UID: \"4af7e1f9-3e4c-4a48-8899-de031c15e1ec\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:02:00.080808 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.080783 6999 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jggd\" (UniqueName: \"kubernetes.io/projected/4af7e1f9-3e4c-4a48-8899-de031c15e1ec-kube-api-access-9jggd\") pod \"perf-node-gather-daemonset-cl252\" (UID: \"4af7e1f9-3e4c-4a48-8899-de031c15e1ec\") " pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:02:00.096881 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.096866 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-wm62c_0e3d6cb4-b70b-4d68-aca8-53fdb36f017d/server/0.log" Oct 30 07:02:00.210490 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.210404 6999 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:02:00.340979 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.340955 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kjczm_c65427c6-ca18-462d-b777-c52b0b195c9c/manager/0.log" Oct 30 07:02:00.380526 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.380499 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wmwrb_4d8da3c6-77af-4a68-aa92-89594aaa5037/proxy/0.log" Oct 30 07:02:00.520684 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.520642 6999 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252"] Oct 30 07:02:00.523433 ip-10-0-131-226 kubenswrapper[6999]: W1030 07:02:00.523406 6999 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4af7e1f9_3e4c_4a48_8899_de031c15e1ec.slice/crio-e8adfa824e4f78cdbb68c846ba38d9899da8e91ec1bf70141ffb1d09046dc9e8 WatchSource:0}: Error finding container e8adfa824e4f78cdbb68c846ba38d9899da8e91ec1bf70141ffb1d09046dc9e8: Status 404 returned error can't find the container with id e8adfa824e4f78cdbb68c846ba38d9899da8e91ec1bf70141ffb1d09046dc9e8 Oct 30 07:02:00.700866 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.700839 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" event={"ID":"4af7e1f9-3e4c-4a48-8899-de031c15e1ec","Type":"ContainerStarted","Data":"b247ca0550a71677d408f6c72314f689418a816e9af7410947bdb9bc8b0acee6"} Oct 30 07:02:00.701185 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.700893 6999 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" event={"ID":"4af7e1f9-3e4c-4a48-8899-de031c15e1ec","Type":"ContainerStarted","Data":"e8adfa824e4f78cdbb68c846ba38d9899da8e91ec1bf70141ffb1d09046dc9e8"} Oct 30 07:02:00.701185 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.700931 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:02:00.742033 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.741992 6999 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" podStartSLOduration=1.741980855 podStartE2EDuration="1.741980855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 07:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 07:02:00.740495331 +0000 UTC m=+4087.249515218" watchObservedRunningTime="2026-10-30 07:02:00.741980855 +0000 UTC m=+4087.251000744" Oct 30 07:02:00.912972 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:00.912948 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dd578_d2d51b73-b344-44d8-8fbf-84b50e45cbe5/manager/0.log" Oct 30 07:02:01.534465 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:01.534432 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5xns7_c2c64f13-36fe-4739-930b-f094dd0e225c/manager/0.log" Oct 30 07:02:01.583937 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:01.583903 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/0.log" Oct 30 07:02:01.782692 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:01.782659 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_678a3a08-9806-4cb8-923f-8f45c7f84869/alertmanager/0.log" Oct 30 07:02:01.861281 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:01.861253 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/1.log" Oct 30 07:02:01.911308 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:01.911279 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_678a3a08-9806-4cb8-923f-8f45c7f84869/config-reloader/0.log" Oct 30 07:02:01.921798 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:01.921775 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-wm62c_0e3d6cb4-b70b-4d68-aca8-53fdb36f017d/server/0.log" Oct 30 07:02:02.027826 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:02.027796 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_678a3a08-9806-4cb8-923f-8f45c7f84869/kube-rbac-proxy-web/0.log" Oct 30 07:02:02.131613 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:02.131534 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kjczm_c65427c6-ca18-462d-b777-c52b0b195c9c/manager/0.log" Oct 30 07:02:02.150862 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:02.150843 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_678a3a08-9806-4cb8-923f-8f45c7f84869/kube-rbac-proxy/0.log" Oct 30 07:02:02.175945 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:02.175919 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wmwrb_4d8da3c6-77af-4a68-aa92-89594aaa5037/proxy/0.log" Oct 30 07:02:02.176547 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:02.176530 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dd578_d2d51b73-b344-44d8-8fbf-84b50e45cbe5/manager/0.log" Oct 30 07:02:02.322099 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:02.322073 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_678a3a08-9806-4cb8-923f-8f45c7f84869/kube-rbac-proxy-metric/0.log" Oct 30 07:02:02.451035 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:02.450951 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_678a3a08-9806-4cb8-923f-8f45c7f84869/prom-label-proxy/0.log" Oct 30 07:02:02.573432 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:02.573396 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_678a3a08-9806-4cb8-923f-8f45c7f84869/init-config-reloader/0.log" Oct 30 07:02:02.823374 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:02.823286 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5nppf_9cb92418-7659-4dad-9ffc-fa4e29f099a4/kube-state-metrics/0.log" Oct 30 07:02:02.833259 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:02.833231 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5xns7_c2c64f13-36fe-4739-930b-f094dd0e225c/manager/0.log" Oct 30 07:02:02.898614 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:02.898579 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/0.log" Oct 30 07:02:02.933129 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:02.933098 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5nppf_9cb92418-7659-4dad-9ffc-fa4e29f099a4/kube-rbac-proxy-main/0.log" Oct 30 07:02:03.054428 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:03.054400 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-5nppf_9cb92418-7659-4dad-9ffc-fa4e29f099a4/kube-rbac-proxy-self/0.log" Oct 30 07:02:03.166878 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:03.166844 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/1.log" Oct 30 07:02:03.180833 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:03.180807 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-755798cb46-949x8_8b944931-8f1e-4596-a265-ceec8c673872/metrics-server/0.log" Oct 30 07:02:03.220459 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:03.220433 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-wm62c_0e3d6cb4-b70b-4d68-aca8-53fdb36f017d/server/0.log" Oct 30 07:02:03.299383 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:03.299348 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-rzpvh_e2a06249-236e-43d7-b485-0d21b72fb442/monitoring-plugin/0.log" Oct 30 07:02:03.417185 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:03.417100 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kjczm_c65427c6-ca18-462d-b777-c52b0b195c9c/manager/0.log" Oct 30 07:02:03.460754 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:03.460726 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wmwrb_4d8da3c6-77af-4a68-aa92-89594aaa5037/proxy/0.log" Oct 30 07:02:03.799379 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:03.799291 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dd578_d2d51b73-b344-44d8-8fbf-84b50e45cbe5/manager/0.log" Oct 30 07:02:03.802273 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:03.802253 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gfv67_a1192599-1c2d-4670-a19f-c4e9faa70c46/node-exporter/0.log" Oct 30 07:02:03.919973 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:03.919947 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gfv67_a1192599-1c2d-4670-a19f-c4e9faa70c46/kube-rbac-proxy/0.log" Oct 30 07:02:04.036795 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:04.036764 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gfv67_a1192599-1c2d-4670-a19f-c4e9faa70c46/init-textfile/0.log" Oct 30 07:02:04.148374 ip-10-0-131-226 kubenswrapper[6999]: E1030 07:02:04.148326 6999 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-j7wwc" containerName="extractor" Oct 30 07:02:04.410960 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:04.410862 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5xns7_c2c64f13-36fe-4739-930b-f094dd0e225c/manager/0.log" Oct 30 07:02:04.471801 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:04.471767 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/0.log" Oct 30 07:02:04.525184 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:04.525154 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-hq6w7_19c2a004-564f-442c-a29a-ebcd20f71102/kube-rbac-proxy-main/0.log" Oct 30 07:02:04.650129 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:04.650092 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-hq6w7_19c2a004-564f-442c-a29a-ebcd20f71102/kube-rbac-proxy-self/0.log" Oct 30 07:02:04.740824 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:04.740742 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/1.log" Oct 30 07:02:04.766730 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:04.766706 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-hq6w7_19c2a004-564f-442c-a29a-ebcd20f71102/openshift-state-metrics/0.log" Oct 30 07:02:04.832583 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:04.832559 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-wm62c_0e3d6cb4-b70b-4d68-aca8-53fdb36f017d/server/0.log" Oct 30 07:02:04.898704 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:04.898677 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e60660bc-2e77-4a26-bf45-beab6a89adc7/prometheus/0.log" Oct 30 07:02:04.968473 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:04.968449 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dd578_d2d51b73-b344-44d8-8fbf-84b50e45cbe5/manager/0.log" Oct 30 07:02:05.015599 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:05.015524 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e60660bc-2e77-4a26-bf45-beab6a89adc7/config-reloader/0.log" Oct 30 07:02:05.057170 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:05.057124 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kjczm_c65427c6-ca18-462d-b777-c52b0b195c9c/manager/0.log" Oct 30 07:02:05.102817 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:05.102789 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wmwrb_4d8da3c6-77af-4a68-aa92-89594aaa5037/proxy/0.log" Oct 30 07:02:05.137764 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:05.137741 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e60660bc-2e77-4a26-bf45-beab6a89adc7/thanos-sidecar/0.log" Oct 30 07:02:05.256906 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:05.256886 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e60660bc-2e77-4a26-bf45-beab6a89adc7/kube-rbac-proxy-web/0.log" Oct 30 07:02:05.374844 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:05.374820 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e60660bc-2e77-4a26-bf45-beab6a89adc7/kube-rbac-proxy/0.log" Oct 30 07:02:05.495649 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:05.495622 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e60660bc-2e77-4a26-bf45-beab6a89adc7/kube-rbac-proxy-thanos/0.log" Oct 30 07:02:05.577761 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:05.577732 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5xns7_c2c64f13-36fe-4739-930b-f094dd0e225c/manager/0.log" Oct 30 07:02:05.611583 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:05.611560 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e60660bc-2e77-4a26-bf45-beab6a89adc7/init-config-reloader/0.log" Oct 30 07:02:05.662645 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:05.662582 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/0.log" Oct 30 07:02:05.958059 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:05.957973 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/1.log" Oct 30 07:02:06.007989 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:06.007961 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-wm62c_0e3d6cb4-b70b-4d68-aca8-53fdb36f017d/server/0.log" Oct 30 07:02:06.212575 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:06.212486 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kjczm_c65427c6-ca18-462d-b777-c52b0b195c9c/manager/0.log" Oct 30 07:02:06.223966 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:06.223920 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7ff58fc747-xt48n_e1db1c23-703c-4455-8bbf-23688bec2ede/telemeter-client/0.log" Oct 30 07:02:06.257170 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:06.257143 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wmwrb_4d8da3c6-77af-4a68-aa92-89594aaa5037/proxy/0.log" Oct 30 07:02:06.341902 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:06.341863 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7ff58fc747-xt48n_e1db1c23-703c-4455-8bbf-23688bec2ede/reload/0.log" Oct 30 07:02:06.462244 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:06.462187 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7ff58fc747-xt48n_e1db1c23-703c-4455-8bbf-23688bec2ede/kube-rbac-proxy/0.log" Oct 30 07:02:06.583924 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:06.583841 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-95d7f5dd9-d9f6f_4e1f5208-4203-46bf-ab5f-8587b9be50de/thanos-query/0.log" Oct 30 07:02:06.653804 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:06.653776 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-dd578_d2d51b73-b344-44d8-8fbf-84b50e45cbe5/manager/0.log" Oct 30 07:02:06.697775 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:06.697756 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-95d7f5dd9-d9f6f_4e1f5208-4203-46bf-ab5f-8587b9be50de/kube-rbac-proxy-web/0.log" Oct 30 07:02:06.715863 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:06.715844 6999 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5nqnt/perf-node-gather-daemonset-cl252" Oct 30 07:02:06.813480 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:06.813443 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-95d7f5dd9-d9f6f_4e1f5208-4203-46bf-ab5f-8587b9be50de/kube-rbac-proxy/0.log" Oct 30 07:02:06.931849 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:06.931822 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-95d7f5dd9-d9f6f_4e1f5208-4203-46bf-ab5f-8587b9be50de/prom-label-proxy/0.log" Oct 30 07:02:07.048286 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:07.048257 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-95d7f5dd9-d9f6f_4e1f5208-4203-46bf-ab5f-8587b9be50de/kube-rbac-proxy-rules/0.log" Oct 30 07:02:07.073721 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:07.073697 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:07.166845 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:07.166805 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-95d7f5dd9-d9f6f_4e1f5208-4203-46bf-ab5f-8587b9be50de/kube-rbac-proxy-metrics/0.log" Oct 30 07:02:07.254712 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:07.254654 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5xns7_c2c64f13-36fe-4739-930b-f094dd0e225c/manager/0.log" Oct 30 07:02:07.373504 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:07.373482 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/0.log" Oct 30 07:02:07.645861 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:07.645836 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5749bffc96-tr2hn_9c87696f-3650-470f-9ddf-2d1137017d59/manager/1.log" Oct 30 07:02:07.707957 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:07.707933 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-wm62c_0e3d6cb4-b70b-4d68-aca8-53fdb36f017d/server/0.log" Oct 30 07:02:07.912007 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:07.911926 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-kjczm_c65427c6-ca18-462d-b777-c52b0b195c9c/manager/0.log" Oct 30 07:02:07.955054 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:07.955025 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-wmwrb_4d8da3c6-77af-4a68-aa92-89594aaa5037/proxy/0.log" Oct 30 07:02:09.544333 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:09.544305 6999 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8mvtx_33bd8834-ad74-4370-b805-a6995dd16230/konnectivity-agent/0.log" limit=1024 Oct 30 07:02:10.822006 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:10.821975 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c4bb65c68-z6847_0d476729-9d6f-4624-b080-e19e28bcd0f5/kube-auth-proxy/0.log" Oct 30 07:02:10.962771 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:10.962743 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-mlqh8_edf39134-4250-4b1c-af52-f5d4f8ddcd10/istio-proxy/0.log" Oct 30 07:02:12.643262 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:12.643235 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c4bb65c68-z6847_0d476729-9d6f-4624-b080-e19e28bcd0f5/kube-auth-proxy/0.log" Oct 30 07:02:12.776287 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:12.776263 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-mlqh8_edf39134-4250-4b1c-af52-f5d4f8ddcd10/istio-proxy/0.log" Oct 30 07:02:15.857311 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:15.857277 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tvz4j_3e9c238a-4ad9-4ad2-98d0-cb4509704ec0/dns/0.log" Oct 30 07:02:15.993880 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:15.993839 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tvz4j_3e9c238a-4ad9-4ad2-98d0-cb4509704ec0/kube-rbac-proxy/0.log" Oct 30 07:02:16.354315 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:16.354276 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zqrdw_1b9d8b00-b8a3-4082-ad89-053a60ff7078/dns-node-resolver/0.log" Oct 30 07:02:17.420746 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:17.420716 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-bwccj_013f3068-30a0-4b4c-b313-6eee3520eb14/kuadrant-console-plugin/0.log" Oct 30 07:02:17.566410 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:17.566373 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c4bb65c68-z6847_0d476729-9d6f-4624-b080-e19e28bcd0f5/kube-auth-proxy/0.log" Oct 30 07:02:17.718021 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:17.717920 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-mlqh8_edf39134-4250-4b1c-af52-f5d4f8ddcd10/istio-proxy/0.log" Oct 30 07:02:17.977557 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:17.977474 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vjrj8_733fbaf7-e75e-46c7-ad59-478b58424087/node-ca/0.log" Oct 30 07:02:18.844818 ip-10-0-131-226 kubenswrapper[6999]: I1030 07:02:18.844792 6999 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-bwccj_013f3068-30a0-4b4c-b313-6eee3520eb14/kuadrant-console-plugin/0.log"